Oct 29 22:41:52.081952 ip-10-0-135-80 systemd[1]: Starting Kubernetes Kubelet... Oct 29 22:41:52.457361 ip-10-0-135-80 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:41:52.457361 ip-10-0-135-80 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 22:41:52.457361 ip-10-0-135-80 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:41:52.457361 ip-10-0-135-80 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:41:52.461245 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461163 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:41:52.461245 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461247 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:41:52.461317 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461253 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:41:52.461317 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461257 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:41:52.461317 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461260 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:41:52.461317 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461263 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:41:52.461317 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461266 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:41:52.461317 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461268 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:41:52.461317 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461271 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:41:52.461317 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461273 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:41:52.461317 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461276 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:41:52.461317 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461279 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:41:52.461317 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461282 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:41:52.461317 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461284 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:41:52.461317 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461287 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:41:52.461317 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461290 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:41:52.461317 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461293 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:41:52.461317 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461296 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:41:52.461317 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461304 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:41:52.461317 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461307 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:41:52.461317 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461309 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:41:52.461317 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461312 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:41:52.461870 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461314 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:41:52.461870 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461317 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:41:52.461870 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461320 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:41:52.461870 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461323 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:41:52.461870 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461326 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:41:52.461870 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461328 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:41:52.461870 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461331 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:41:52.461870 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461334 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:41:52.461870 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461336 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:41:52.461870 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461339 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:41:52.461870 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461342 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:41:52.461870 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461345 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:41:52.461870 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461347 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:41:52.461870 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461350 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:41:52.461870 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461352 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:41:52.461870 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461356 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:41:52.461870 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461359 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:41:52.461870 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461361 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:41:52.461870 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461363 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:41:52.462305 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461366 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:41:52.462305 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461368 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:41:52.462305 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461371 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:41:52.462305 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461374 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:41:52.462305 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461376 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:41:52.462305 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461379 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:41:52.462305 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461381 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:41:52.462305 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461383 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:41:52.462305 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461386 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:41:52.462305 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461389 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:41:52.462305 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461392 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:41:52.462305 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461395 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:41:52.462305 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461397 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:41:52.462305 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461400 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:41:52.462305 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461403 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:41:52.462305 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461405 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:41:52.462305 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461408 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:41:52.462305 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461411 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:41:52.462305 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461413 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:41:52.462757 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461415 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:41:52.462757 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461418 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:41:52.462757 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461420 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:41:52.462757 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461423 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:41:52.462757 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461425 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:41:52.462757 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461428 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:41:52.462757 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461430 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:41:52.462757 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461433 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:41:52.462757 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461435 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:41:52.462757 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461438 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:41:52.462757 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461440 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:41:52.462757 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461442 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:41:52.462757 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461445 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:41:52.462757 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461447 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:41:52.462757 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461450 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:41:52.462757 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461452 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:41:52.462757 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461455 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:41:52.462757 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461457 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:41:52.462757 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461460 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:41:52.462757 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461462 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:41:52.463220 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461465 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:41:52.463220 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461469 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:41:52.463220 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461474 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:41:52.463220 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461477 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:41:52.463220 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461482 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:41:52.463220 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461484 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:41:52.463220 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461487 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:41:52.463220 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461490 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:41:52.463220 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461493 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:41:52.463220 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461496 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:41:52.463220 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461499 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:41:52.463220 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461502 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:41:52.463220 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461504 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:41:52.463220 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461507 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:41:52.463220 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461509 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:41:52.463220 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461512 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:41:52.463220 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461514 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:41:52.463220 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461517 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:41:52.463220 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461519 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:41:52.463696 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461522 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:41:52.463696 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461524 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:41:52.463696 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461527 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:41:52.463696 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461529 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:41:52.463696 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461532 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:41:52.463696 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461534 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:41:52.463696 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.461537 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:41:52.463696 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462343 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:41:52.463696 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462351 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:41:52.463696 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462355 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:41:52.463696 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462358 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:41:52.463696 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462360 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:41:52.463696 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462363 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:41:52.463696 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462365 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:41:52.463696 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462368 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:41:52.463696 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462371 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:41:52.463696 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462374 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:41:52.463696 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462377 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:41:52.463696 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462380 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:41:52.463696 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462383 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:41:52.464187 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462386 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:41:52.464187 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462388 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:41:52.464187 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462391 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:41:52.464187 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462394 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:41:52.464187 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462396 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:41:52.464187 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462399 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:41:52.464187 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462401 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:41:52.464187 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462404 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:41:52.464187 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462406 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:41:52.464187 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462409 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:41:52.464187 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462411 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:41:52.464187 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462414 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:41:52.464187 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462416 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:41:52.464187 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462419 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:41:52.464187 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462421 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:41:52.464187 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462424 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:41:52.464187 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462427 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:41:52.464187 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462429 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:41:52.464187 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462432 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:41:52.464187 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462435 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:41:52.464650 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462437 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:41:52.464650 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462440 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:41:52.464650 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462443 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:41:52.464650 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462445 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:41:52.464650 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462448 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:41:52.464650 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462450 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:41:52.464650 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462453 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:41:52.464650 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462455 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:41:52.464650 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462458 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:41:52.464650 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462460 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:41:52.464650 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462463 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:41:52.464650 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462465 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:41:52.464650 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462468 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:41:52.464650 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462471 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:41:52.464650 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462473 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:41:52.464650 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462476 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:41:52.464650 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462478 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:41:52.464650 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462481 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:41:52.464650 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462484 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:41:52.464650 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462486 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:41:52.465127 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462489 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:41:52.465127 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462491 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:41:52.465127 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462494 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:41:52.465127 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462496 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:41:52.465127 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462498 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:41:52.465127 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462501 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:41:52.465127 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462504 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:41:52.465127 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462508 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:41:52.465127 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462513 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:41:52.465127 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462518 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:41:52.465127 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462522 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:41:52.465127 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462525 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:41:52.465127 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462528 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:41:52.465127 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462530 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:41:52.465127 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462533 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:41:52.465127 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462536 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:41:52.465127 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462538 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:41:52.465127 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462541 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:41:52.465127 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462543 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:41:52.465571 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462546 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:41:52.465571 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462548 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:41:52.465571 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462551 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:41:52.465571 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462554 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:41:52.465571 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462556 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:41:52.465571 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462560 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:41:52.465571 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462563 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:41:52.465571 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462565 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:41:52.465571 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462568 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:41:52.465571 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462571 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:41:52.465571 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462574 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:41:52.465571 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462576 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:41:52.465571 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462579 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:41:52.465571 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462581 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:41:52.465571 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462583 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:41:52.465571 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462586 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:41:52.465571 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462589 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:41:52.465571 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462591 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:41:52.465571 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462594 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:41:52.466024 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462596 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:41:52.466024 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462599 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:41:52.466024 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462615 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:41:52.466024 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462619 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:41:52.466024 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462622 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:41:52.466024 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462625 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:41:52.466024 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462628 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:41:52.466024 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462631 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:41:52.466024 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462634 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:41:52.466024 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462636 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:41:52.466024 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462639 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:41:52.466024 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462641 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:41:52.466024 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462644 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:41:52.466024 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462646 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:41:52.466024 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.462649 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:41:52.466024 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463151 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:41:52.466024 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463157 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:41:52.466024 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463160 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:41:52.466024 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463164 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:41:52.466464 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463167 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:41:52.466464 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463171 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:41:52.466464 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463173 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:41:52.466464 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463176 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:41:52.466464 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463179 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:41:52.466464 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463181 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:41:52.466464 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463184 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:41:52.466464 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463186 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:41:52.466464 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463189 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:41:52.466464 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463192 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:41:52.466464 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463194 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:41:52.466464 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463197 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:41:52.466464 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463199 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:41:52.466464 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463203 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:41:52.466464 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463207 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:41:52.466464 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463210 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:41:52.466464 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463213 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:41:52.466464 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463216 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:41:52.466464 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463219 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:41:52.466916 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463221 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:41:52.466916 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463224 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:41:52.466916 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463227 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:41:52.466916 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463229 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:41:52.466916 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463232 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:41:52.466916 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463235 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:41:52.466916 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463237 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:41:52.466916 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463240 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:41:52.466916 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463242 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:41:52.466916 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463245 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:41:52.466916 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463248 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:41:52.466916 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463252 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:41:52.466916 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463255 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:41:52.466916 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463258 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:41:52.466916 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463261 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:41:52.466916 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463264 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:41:52.466916 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463266 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:41:52.466916 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463269 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:41:52.466916 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463271 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:41:52.466916 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463274 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:41:52.467376 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463277 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:41:52.467376 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463280 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:41:52.467376 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463283 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:41:52.467376 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463285 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:41:52.467376 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463288 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:41:52.467376 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463290 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:41:52.467376 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463293 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:41:52.467376 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463295 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:41:52.467376 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463298 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:41:52.467376 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463301 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:41:52.467376 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463303 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:41:52.467376 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463306 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:41:52.467376 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463308 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:41:52.467376 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463311 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:41:52.467376 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463314 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:41:52.467376 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463316 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:41:52.467376 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463319 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:41:52.467376 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463321 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:41:52.467376 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463324 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:41:52.467376 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463327 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:41:52.467874 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463329 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:41:52.467874 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463332 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:41:52.467874 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463334 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:41:52.467874 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463337 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:41:52.467874 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463339 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:41:52.467874 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463342 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:41:52.467874 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463345 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:41:52.467874 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463348 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:41:52.467874 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463350 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:41:52.467874 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463353 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:41:52.467874 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463355 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:41:52.467874 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463358 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:41:52.467874 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463360 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:41:52.467874 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463363 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:41:52.467874 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463365 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:41:52.467874 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463367 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:41:52.467874 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463370 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:41:52.467874 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463373 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:41:52.467874 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463375 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:41:52.468326 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463378 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:41:52.468326 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463380 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:41:52.468326 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463383 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:41:52.468326 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463386 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:41:52.468326 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463388 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:41:52.468326 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463391 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:41:52.468326 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463393 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:41:52.468326 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463395 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:41:52.468326 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463398 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:41:52.468326 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463400 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:41:52.468326 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463403 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:41:52.468326 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463405 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:41:52.468326 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463408 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:41:52.468326 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463410 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:41:52.468326 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463413 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:41:52.468326 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463415 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:41:52.468326 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463418 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:41:52.468326 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463422 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:41:52.468326 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463425 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:41:52.468326 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463427 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463434 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463437 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463439 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.463443 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465439 2581 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465449 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465459 2581 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465465 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465469 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465473 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465478 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465483 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465486 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465490 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465493 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465497 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465500 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465503 2581 flags.go:64] FLAG: --cgroup-root="" Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465506 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465509 2581 flags.go:64] FLAG: --client-ca-file="" Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465512 2581 flags.go:64] FLAG: --cloud-provider="external" Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465515 2581 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 22:41:52.468796 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465522 2581 flags.go:64] FLAG: --cluster-domain="" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465525 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465528 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465532 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465535 2581 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465539 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465542 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465547 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465550 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465553 2581 flags.go:64] FLAG: --contention-profiling="false" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465557 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465560 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465564 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465567 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465571 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465574 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465577 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465580 2581 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465584 2581 flags.go:64] FLAG: --enable-server="true" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465587 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465591 2581 flags.go:64] FLAG: --event-burst="100" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465594 2581 flags.go:64] FLAG: --event-qps="50" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465597 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465600 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 22:41:52.469354 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465603 2581 flags.go:64] FLAG: --eviction-hard="" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465608 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465611 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465614 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465617 2581 flags.go:64] FLAG: --eviction-soft="" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465620 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465623 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465626 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465629 2581 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465632 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465635 2581 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465638 2581 flags.go:64] FLAG: --feature-gates="" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465642 2581 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465645 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465648 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465652 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465655 2581 flags.go:64] FLAG: --healthz-port="10248" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465659 2581 flags.go:64] FLAG: --help="false" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465662 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-135-80.ec2.internal" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465675 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465679 2581 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465682 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465685 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 22:41:52.469980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465689 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465692 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465695 2581 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465698 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465701 2581 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465704 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465707 2581 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465710 2581 flags.go:64] FLAG: --kube-reserved="" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465713 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465716 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465719 2581 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465721 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465724 2581 flags.go:64] FLAG: --lock-file="" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465727 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465730 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465733 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465739 2581 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465741 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465745 2581 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465748 2581 flags.go:64] FLAG: --logging-format="text" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465750 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465754 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465756 2581 flags.go:64] FLAG: --manifest-url="" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465759 2581 flags.go:64] FLAG: --manifest-url-header="" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465764 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 22:41:52.470520 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465767 2581 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465771 2581 flags.go:64] FLAG: --max-pods="110" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465774 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465777 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465780 2581 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465783 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465787 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465790 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465793 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465801 2581 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465804 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465807 2581 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465810 2581 flags.go:64] FLAG: --pod-cidr="" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465813 2581 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465816 2581 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465819 2581 flags.go:64] FLAG: --pods-per-core="0" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465822 2581 flags.go:64] FLAG: --port="10250" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465825 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465828 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01971e8024657c89d" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465831 2581 flags.go:64] FLAG: --qos-reserved="" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465834 2581 flags.go:64] FLAG: --read-only-port="10255" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465837 2581 flags.go:64] FLAG: --register-node="true" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465840 2581 flags.go:64] FLAG: --register-with-taints="" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465844 2581 flags.go:64] FLAG: --registry-burst="10" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465847 2581 flags.go:64] FLAG: --registry-qps="5" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465850 2581 flags.go:64] FLAG: --reserved-cpus="" Oct 29 22:41:52.471134 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465853 2581 flags.go:64] FLAG: --reserved-memory="" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465857 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465860 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465863 2581 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465866 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465871 2581 flags.go:64] FLAG: --runonce="false" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465874 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465877 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465880 2581 flags.go:64] FLAG: --seccomp-default="false" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465883 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465886 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465889 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465894 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465897 2581 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465900 2581 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465903 2581 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465906 2581 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465909 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465913 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465916 2581 flags.go:64] FLAG: --system-cgroups="" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465918 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465924 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465927 2581 flags.go:64] FLAG: --tls-cert-file="" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465930 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465934 2581 flags.go:64] FLAG: --tls-min-version="" Oct 29 22:41:52.471784 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465937 2581 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465940 2581 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465943 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465945 2581 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465949 2581 flags.go:64] FLAG: --v="2" Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465953 2581 flags.go:64] FLAG: --version="false" Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465957 2581 flags.go:64] FLAG: --vmodule="" Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465961 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.465965 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466102 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466105 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466108 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466113 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466116 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466119 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466122 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466125 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466128 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466131 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466134 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466137 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466139 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:41:52.472750 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466142 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:41:52.473409 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466145 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:41:52.473409 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466147 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:41:52.473409 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466150 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:41:52.473409 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466153 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:41:52.473409 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466156 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:41:52.473409 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466158 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:41:52.473409 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466161 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:41:52.473409 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466163 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:41:52.473409 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466166 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:41:52.473409 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466169 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:41:52.473409 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466171 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:41:52.473409 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466174 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:41:52.473409 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466176 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:41:52.473409 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466179 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:41:52.473409 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466182 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:41:52.473409 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466184 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:41:52.473409 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466187 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:41:52.473409 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466189 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:41:52.473409 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466192 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:41:52.473863 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466195 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:41:52.473863 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466197 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:41:52.473863 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466203 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:41:52.473863 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466208 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:41:52.473863 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466211 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:41:52.473863 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466214 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:41:52.473863 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466217 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:41:52.473863 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466219 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:41:52.473863 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466222 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:41:52.473863 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466225 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:41:52.473863 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466227 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:41:52.473863 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466230 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:41:52.473863 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466233 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:41:52.473863 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466236 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:41:52.473863 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466238 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:41:52.473863 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466241 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:41:52.473863 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466243 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:41:52.473863 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466246 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:41:52.473863 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466248 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:41:52.474332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466251 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:41:52.474332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466254 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:41:52.474332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466256 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:41:52.474332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466259 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:41:52.474332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466261 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:41:52.474332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466264 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:41:52.474332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466266 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:41:52.474332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466269 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:41:52.474332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466271 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:41:52.474332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466274 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:41:52.474332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466276 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:41:52.474332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466279 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:41:52.474332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466282 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:41:52.474332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466285 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:41:52.474332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466288 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:41:52.474332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466294 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:41:52.474332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466298 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:41:52.474332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466301 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:41:52.474779 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466303 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:41:52.474779 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466306 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:41:52.474779 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466309 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:41:52.474779 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466311 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:41:52.474779 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466314 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:41:52.474779 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466317 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:41:52.474779 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466319 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:41:52.474779 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466322 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:41:52.474779 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466325 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:41:52.474779 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466328 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:41:52.474779 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466330 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:41:52.474779 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466333 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:41:52.474779 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466336 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:41:52.474779 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466338 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:41:52.474779 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466341 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:41:52.474779 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466343 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:41:52.474779 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466346 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:41:52.474779 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466349 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:41:52.474779 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466351 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:41:52.474779 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466354 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:41:52.475239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466356 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:41:52.475239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466359 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:41:52.475239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466361 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:41:52.475239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466364 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:41:52.475239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466366 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:41:52.475239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466369 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:41:52.475239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466371 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:41:52.475239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466374 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:41:52.475239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466376 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:41:52.475239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466382 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:41:52.475239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466385 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:41:52.475239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466387 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:41:52.475239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466390 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:41:52.475239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466392 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:41:52.475239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466395 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:41:52.475239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466398 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:41:52.475621 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.466425 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:41:52.475621 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466460 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:41:52.475621 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466464 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:41:52.475621 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466467 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:41:52.475621 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466471 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:41:52.475621 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466474 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:41:52.475621 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466477 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:41:52.475621 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466480 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:41:52.475621 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466482 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:41:52.475621 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466485 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:41:52.475621 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466488 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:41:52.475621 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466490 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:41:52.475621 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466493 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:41:52.475621 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466496 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:41:52.475621 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466498 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:41:52.475621 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466501 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:41:52.476018 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466503 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:41:52.476018 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466506 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:41:52.476018 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466508 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:41:52.476018 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466511 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:41:52.476018 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466513 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:41:52.476018 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466516 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:41:52.476018 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466519 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:41:52.476018 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466521 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:41:52.476018 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466525 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:41:52.476018 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466530 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:41:52.476018 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466537 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:41:52.476018 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466540 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:41:52.476018 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466542 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:41:52.476018 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466545 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:41:52.476018 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466548 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:41:52.476018 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466551 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:41:52.476018 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466553 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:41:52.476018 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466556 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:41:52.476018 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466559 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:41:52.476475 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466561 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:41:52.476475 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466564 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:41:52.476475 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466567 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:41:52.476475 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466570 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:41:52.476475 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466572 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:41:52.476475 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466575 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:41:52.476475 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466578 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:41:52.476475 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466580 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:41:52.476475 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466582 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:41:52.476475 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466585 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:41:52.476475 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466588 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:41:52.476475 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466590 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:41:52.476475 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466593 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:41:52.476475 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466595 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:41:52.476475 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466598 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:41:52.476475 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466600 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:41:52.476475 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466603 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:41:52.476475 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466606 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:41:52.476475 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466608 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:41:52.476936 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466611 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:41:52.476936 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466614 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:41:52.476936 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466618 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:41:52.476936 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466623 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:41:52.476936 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466627 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:41:52.476936 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466630 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:41:52.476936 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466632 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:41:52.476936 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466635 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:41:52.476936 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466637 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:41:52.476936 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466640 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:41:52.476936 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466643 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:41:52.476936 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466645 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:41:52.476936 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466648 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:41:52.476936 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466651 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:41:52.476936 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466653 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:41:52.476936 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466656 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:41:52.476936 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466659 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:41:52.476936 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466662 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:41:52.476936 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466676 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:41:52.477378 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466679 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:41:52.477378 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466682 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:41:52.477378 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466684 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:41:52.477378 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466687 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:41:52.477378 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466689 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:41:52.477378 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466692 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:41:52.477378 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466695 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:41:52.477378 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466697 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:41:52.477378 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466700 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:41:52.477378 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466702 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:41:52.477378 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466705 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:41:52.477378 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466708 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:41:52.477378 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466710 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:41:52.477378 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466713 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:41:52.477378 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466715 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:41:52.477378 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466718 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:41:52.477378 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466722 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:41:52.477378 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466726 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:41:52.477378 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466728 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:41:52.477378 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466731 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:41:52.477880 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466733 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:41:52.477880 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466737 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:41:52.477880 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466739 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:41:52.477880 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466742 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:41:52.477880 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466744 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:41:52.477880 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466747 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:41:52.477880 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466750 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:41:52.477880 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466752 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:41:52.477880 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466755 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:41:52.477880 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466758 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:41:52.477880 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466761 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:41:52.477880 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466765 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:41:52.477880 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466768 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:41:52.477880 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.466770 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:41:52.477880 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.474298 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 22:41:52.477880 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.474314 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 22:41:52.477880 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.474331 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 22:41:52.477880 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.474336 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 22:41:52.477880 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474396 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:41:52.477880 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474401 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:41:52.478327 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474405 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:41:52.478327 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474408 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:41:52.478327 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474411 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:41:52.478327 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474413 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:41:52.478327 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474416 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:41:52.478327 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474420 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:41:52.478327 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474423 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:41:52.478327 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474426 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:41:52.478327 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474431 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:41:52.478327 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474436 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:41:52.478327 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474439 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:41:52.478327 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474443 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:41:52.478327 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474446 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:41:52.478327 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474452 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:41:52.478327 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474455 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:41:52.478327 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474458 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:41:52.478327 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474460 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:41:52.478327 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474463 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:41:52.478327 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474466 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:41:52.478773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474468 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:41:52.478773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474471 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:41:52.478773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474474 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:41:52.478773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474477 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:41:52.478773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474479 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:41:52.478773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474482 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:41:52.478773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474484 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:41:52.478773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474487 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:41:52.478773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474490 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:41:52.478773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474493 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:41:52.478773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474495 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:41:52.478773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474498 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:41:52.478773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474500 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:41:52.478773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474503 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:41:52.478773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474506 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:41:52.478773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474508 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:41:52.478773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474511 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:41:52.478773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474513 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:41:52.478773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474516 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:41:52.478773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474519 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:41:52.479233 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474521 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:41:52.479233 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474524 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:41:52.479233 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474527 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:41:52.479233 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474530 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:41:52.479233 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474532 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:41:52.479233 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474536 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:41:52.479233 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474538 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:41:52.479233 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474541 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:41:52.479233 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474543 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:41:52.479233 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474546 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:41:52.479233 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474549 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:41:52.479233 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474551 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:41:52.479233 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474554 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:41:52.479233 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474557 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:41:52.479233 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474559 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:41:52.479233 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474562 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:41:52.479233 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474564 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:41:52.479233 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474567 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:41:52.479233 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474569 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:41:52.479834 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474572 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:41:52.479834 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474576 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:41:52.479834 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474579 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:41:52.479834 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474582 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:41:52.479834 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474585 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:41:52.479834 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474588 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:41:52.479834 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474591 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:41:52.479834 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474594 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:41:52.479834 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474596 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:41:52.479834 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474599 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:41:52.479834 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474601 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:41:52.479834 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474604 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:41:52.479834 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474606 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:41:52.479834 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474609 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:41:52.479834 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474612 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:41:52.479834 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474615 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:41:52.479834 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474618 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:41:52.479834 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474620 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:41:52.479834 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474623 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:41:52.479834 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474626 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:41:52.480753 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474629 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:41:52.480753 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474632 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:41:52.480753 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474635 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:41:52.480753 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474637 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:41:52.480753 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474640 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:41:52.480753 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474642 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:41:52.480753 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474645 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:41:52.480753 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474647 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:41:52.480753 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474650 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:41:52.480753 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474652 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:41:52.480753 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474655 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:41:52.480753 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474657 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:41:52.480753 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474660 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:41:52.480753 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474662 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:41:52.480753 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474677 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:41:52.480753 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474679 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:41:52.480753 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474682 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:41:52.480753 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474684 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:41:52.480753 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474687 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:41:52.480753 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474689 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:41:52.481213 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474692 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:41:52.481213 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474694 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:41:52.481213 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474697 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:41:52.481213 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474700 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:41:52.481213 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474702 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:41:52.481213 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474705 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:41:52.481213 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.474733 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:41:52.481213 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474905 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:41:52.481213 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474910 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:41:52.481213 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474913 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:41:52.481213 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474916 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:41:52.481213 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474919 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:41:52.481213 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474922 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:41:52.481213 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474925 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:41:52.481213 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474927 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:41:52.481213 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474930 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:41:52.481574 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474932 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:41:52.481574 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474935 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:41:52.481574 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474937 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:41:52.481574 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474940 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:41:52.481574 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474943 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:41:52.481574 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474945 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:41:52.481574 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474948 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:41:52.481574 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474950 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:41:52.481574 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474953 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:41:52.481574 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474955 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:41:52.481574 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474958 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:41:52.481574 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474960 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:41:52.481574 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474963 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:41:52.481574 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474965 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:41:52.481574 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474968 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:41:52.481574 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474971 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:41:52.481574 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474973 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:41:52.481574 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474976 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:41:52.481574 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474978 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:41:52.481574 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474981 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:41:52.482045 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474983 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:41:52.482045 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474986 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:41:52.482045 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474988 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:41:52.482045 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474991 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:41:52.482045 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474994 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:41:52.482045 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474996 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:41:52.482045 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.474999 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:41:52.482045 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475001 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:41:52.482045 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475004 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:41:52.482045 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475007 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:41:52.482045 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475010 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:41:52.482045 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475013 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:41:52.482045 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475015 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:41:52.482045 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475019 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:41:52.482045 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475021 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:41:52.482045 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475024 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:41:52.482045 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475026 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:41:52.482045 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475028 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:41:52.482045 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475031 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:41:52.482482 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475033 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:41:52.482482 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475036 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:41:52.482482 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475038 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:41:52.482482 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475041 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:41:52.482482 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475043 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:41:52.482482 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475046 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:41:52.482482 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475048 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:41:52.482482 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475051 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:41:52.482482 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475053 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:41:52.482482 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475056 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:41:52.482482 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475058 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:41:52.482482 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475061 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:41:52.482482 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475063 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:41:52.482482 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475066 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:41:52.482482 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475068 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:41:52.482482 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475073 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:41:52.482482 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475076 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:41:52.482482 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475078 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:41:52.482482 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475081 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:41:52.482482 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475083 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:41:52.482953 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475086 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:41:52.482953 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475088 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:41:52.482953 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475092 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:41:52.482953 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475096 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:41:52.482953 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475100 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:41:52.482953 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475103 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:41:52.482953 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475105 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:41:52.482953 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475108 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:41:52.482953 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475110 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:41:52.482953 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475113 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:41:52.482953 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475115 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:41:52.482953 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475118 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:41:52.482953 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475121 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:41:52.482953 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475124 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:41:52.482953 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475128 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:41:52.482953 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475131 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:41:52.482953 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475134 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:41:52.482953 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475136 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:41:52.482953 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475139 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:41:52.483394 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475141 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:41:52.483394 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475144 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:41:52.483394 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475146 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:41:52.483394 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475149 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:41:52.483394 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475151 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:41:52.483394 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475154 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:41:52.483394 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475156 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:41:52.483394 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475159 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:41:52.483394 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475163 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:41:52.483394 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475165 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:41:52.483394 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475168 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:41:52.483394 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475170 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:41:52.483394 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475173 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:41:52.483394 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475175 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:41:52.483394 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475178 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:41:52.483394 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475180 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:41:52.483394 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475183 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:41:52.483394 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475186 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:41:52.483394 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475189 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:41:52.483924 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.475211 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:41:52.483924 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475243 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:41:52.483924 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475248 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:41:52.483924 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475251 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:41:52.483924 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475254 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:41:52.483924 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475256 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:41:52.483924 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475259 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:41:52.483924 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475262 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:41:52.483924 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475266 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:41:52.483924 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475269 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:41:52.483924 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475271 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:41:52.483924 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475274 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:41:52.483924 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475277 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:41:52.483924 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475279 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:41:52.483924 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475282 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:41:52.483924 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475284 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:41:52.483924 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475287 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:41:52.484332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475290 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:41:52.484332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475292 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:41:52.484332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475295 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:41:52.484332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475297 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:41:52.484332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475300 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:41:52.484332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475303 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:41:52.484332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475306 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:41:52.484332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475309 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:41:52.484332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475311 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:41:52.484332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475314 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:41:52.484332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475316 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:41:52.484332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475319 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:41:52.484332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475322 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:41:52.484332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475324 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:41:52.484332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475327 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:41:52.484332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475330 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:41:52.484332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475332 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:41:52.484332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475335 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:41:52.484332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475337 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:41:52.484776 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475340 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:41:52.484776 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475342 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:41:52.484776 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475345 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:41:52.484776 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475347 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:41:52.484776 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475351 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:41:52.484776 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475354 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:41:52.484776 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475356 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:41:52.484776 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475359 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:41:52.484776 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475362 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:41:52.484776 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475364 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:41:52.484776 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475367 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:41:52.484776 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475370 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:41:52.484776 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475372 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:41:52.484776 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475375 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:41:52.484776 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475377 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:41:52.484776 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475380 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:41:52.484776 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475383 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:41:52.484776 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475385 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:41:52.484776 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475388 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:41:52.484776 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475390 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:41:52.485239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475393 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:41:52.485239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475395 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:41:52.485239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475398 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:41:52.485239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475400 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:41:52.485239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475403 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:41:52.485239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475406 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:41:52.485239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475408 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:41:52.485239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475411 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:41:52.485239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475414 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:41:52.485239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475417 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:41:52.485239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475420 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:41:52.485239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475422 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:41:52.485239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475425 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:41:52.485239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475427 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:41:52.485239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475430 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:41:52.485239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475432 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:41:52.485239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475436 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:41:52.485239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475439 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:41:52.485239 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475442 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:41:52.485721 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475445 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:41:52.485721 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475448 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:41:52.485721 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475451 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:41:52.485721 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475453 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:41:52.485721 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475456 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:41:52.485721 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475458 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:41:52.485721 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475461 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:41:52.485721 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475464 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:41:52.485721 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475466 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:41:52.485721 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475469 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:41:52.485721 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475471 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:41:52.485721 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475474 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:41:52.485721 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475477 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:41:52.485721 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475479 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:41:52.485721 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475482 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:41:52.485721 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475484 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:41:52.485721 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475487 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:41:52.485721 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475490 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:41:52.485721 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475494 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:41:52.486178 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475497 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:41:52.486178 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475500 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:41:52.486178 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475503 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:41:52.486178 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475506 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:41:52.486178 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475509 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:41:52.486178 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475511 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:41:52.486178 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475514 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:41:52.486178 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475516 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:41:52.486178 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475519 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:41:52.486178 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475522 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:41:52.486178 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475527 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:41:52.486178 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475530 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:41:52.486178 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:52.475533 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:41:52.486178 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.475590 2581 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 22:41:52.486178 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.478752 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 29 22:41:52.486178 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.479601 2581 server.go:1014] "Starting client certificate rotation" Oct 29 22:41:52.486178 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.479707 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:41:52.486178 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.479743 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:41:52.486615 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.481404 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:41:52.486615 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.483431 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:41:52.492315 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.492297 2581 log.go:25] "Validated CRI v1 runtime API" Oct 29 22:41:52.495511 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.495498 2581 log.go:25] "Validated CRI v1 image API" Oct 29 22:41:52.496338 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.496311 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 22:41:52.499048 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.499025 2581 fs.go:135] Filesystem UUIDs: map[1ecb3b4d-13ea-4249-8958-3396da766a7a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ac1434ae-05a1-4400-8478-7443405f9dc4:/dev/nvme0n1p4] Oct 29 22:41:52.499107 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.499042 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 29 22:41:52.504380 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.504258 2581 manager.go:217] Machine: {Timestamp:2026-10-29 22:41:52.50309002 +0000 UTC m=+0.316744510 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3030108 MemoryCapacity:33164320768 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c00acace8fc6bc6c7eede4040af50 SystemUUID:ec2c00ac-ace8-fc6b-c6c7-eede4040af50 BootID:f0329396-7fbc-4bfb-8f0c-8549fdfce190 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582160384 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582160384 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ff:c7:45:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ff:c7:45:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:8c:a1:a7:15:01 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164320768 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 22:41:52.504380 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.504377 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 22:41:52.504498 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.504454 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 22:41:52.506147 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.506121 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 22:41:52.506292 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.506148 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-80.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 22:41:52.506331 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.506297 2581 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 22:41:52.506331 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.506304 2581 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 22:41:52.506388 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.506340 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:41:52.506388 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.506363 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:41:52.507098 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.507088 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 22:41:52.510067 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.510049 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 22:41:52.510895 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.510866 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 22:41:52.511531 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.511513 2581 kubelet.go:493] "Attempting to sync node with API server" Oct 29 22:41:52.511603 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.511539 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 22:41:52.511603 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.511555 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 22:41:52.511603 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.511569 2581 kubelet.go:405] "Adding apiserver pod source" Oct 29 22:41:52.511603 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.511591 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 22:41:52.512114 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.512095 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:41:52.514652 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.514634 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 22:41:52.519083 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.519070 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:41:52.519812 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.519799 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 22:41:52.519859 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.519829 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 22:41:52.519990 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.519981 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 22:41:52.520019 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.519996 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 22:41:52.520019 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.520003 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 22:41:52.520019 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.520008 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 22:41:52.520019 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.520014 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 22:41:52.520019 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.520020 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 22:41:52.520532 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.520522 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 22:41:52.520559 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.520532 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 22:41:52.520559 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.520539 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 22:41:52.520559 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.520545 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 22:41:52.520559 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.520554 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 22:41:52.520773 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.520563 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 22:41:52.521162 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.521153 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 22:41:52.521191 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.521163 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 22:41:52.524969 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.524957 2581 server.go:1263] "Started kubelet" Oct 29 22:41:52.525111 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.525074 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 22:41:52.525255 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.525043 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 22:41:52.525322 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.525280 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 22:41:52.525792 ip-10-0-135-80 systemd[1]: Started Kubernetes Kubelet. Oct 29 22:41:52.526882 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.526860 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 22:41:52.527701 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.527685 2581 server.go:317] "Adding debug handlers to kubelet server" Oct 29 22:41:52.531078 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.531059 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 22:41:52.532097 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.532078 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 22:41:52.532470 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:52.532448 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-80.ec2.internal\" not found" Oct 29 22:41:52.532676 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:52.532635 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 29 22:41:52.532916 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.532902 2581 factory.go:55] Registering systemd factory Oct 29 22:41:52.532916 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.532917 2581 factory.go:223] Registration of the systemd container factory successfully Oct 29 22:41:52.533140 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.533125 2581 factory.go:153] Registering CRI-O factory Oct 29 22:41:52.533184 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.533144 2581 factory.go:223] Registration of the crio container factory successfully Oct 29 22:41:52.533212 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.533203 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 22:41:52.533241 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.533228 2581 factory.go:103] Registering Raw factory Oct 29 22:41:52.533267 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.533244 2581 manager.go:1199] Started watching for new ooms in manager Oct 29 22:41:52.533886 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.533858 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 22:41:52.533886 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.533859 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 22:41:52.533886 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.533885 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 22:41:52.534090 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.534078 2581 reconstruct.go:97] "Volume reconstruction finished" Oct 29 22:41:52.534090 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.534091 2581 reconciler.go:29] "Reconciler: start to sync state" Oct 29 22:41:52.534531 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.534506 2581 manager.go:319] Starting recovery of all containers Oct 29 22:41:52.544753 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.544569 2581 manager.go:324] Recovery completed Oct 29 22:41:52.549805 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.549135 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:41:52.551059 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.551041 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:41:52.551129 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.551075 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:41:52.551129 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.551086 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:41:52.551644 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.551630 2581 cpu_manager.go:226] "Starting" policy="none" Oct 29 22:41:52.551713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.551644 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 22:41:52.551713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.551650 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:41:52.551713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.551680 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 22:41:52.552405 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.552351 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:41:52.552405 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.552377 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:41:52.552405 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.552390 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:41:52.552989 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.552977 2581 policy_none.go:50] "Start" Oct 29 22:41:52.553031 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.553000 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 22:41:52.553031 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.553009 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 22:41:52.553980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.553965 2581 policy_none.go:44] "Start" Oct 29 22:41:52.558878 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:52.558852 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 29 22:41:52.558972 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.558928 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-80.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 29 22:41:52.559909 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:52.558965 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-80.ec2.internal.18e32151f5e2b1d7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-80.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-80.ec2.internal,},FirstTimestamp:2026-10-29 22:41:52.524931543 +0000 UTC m=+0.338586031,LastTimestamp:2026-10-29 22:41:52.524931543 +0000 UTC m=+0.338586031,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-80.ec2.internal,}" Oct 29 22:41:52.575017 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.574998 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 22:41:52.582127 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:52.582107 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 29 22:41:52.582221 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:52.582104 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-80.ec2.internal.18e32151f771621a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-80.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-80.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-80.ec2.internal,},FirstTimestamp:2026-10-29 22:41:52.551059994 +0000 UTC m=+0.364714489,LastTimestamp:2026-10-29 22:41:52.551059994 +0000 UTC m=+0.364714489,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-80.ec2.internal,}" Oct 29 22:41:52.582306 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:52.582169 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 29 22:41:52.584199 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.584172 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 22:41:52.586541 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.586529 2581 manager.go:341] "Starting Device Plugin manager" Oct 29 22:41:52.586598 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:52.586555 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 22:41:52.586598 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.586563 2581 server.go:87] "Starting device plugin registration server" Oct 29 22:41:52.586830 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.586814 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 22:41:52.586898 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.586828 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 22:41:52.586945 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.586908 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 22:41:52.587010 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.586992 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 22:41:52.587010 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.587007 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 22:41:52.587456 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:52.587439 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 22:41:52.587521 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:52.587485 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-80.ec2.internal\" not found" Oct 29 22:41:52.593733 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:52.593711 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 29 22:41:52.604108 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:52.604035 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-80.ec2.internal.18e32151f771b2b6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-80.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-80.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-80.ec2.internal,},FirstTimestamp:2026-10-29 22:41:52.55108063 +0000 UTC m=+0.364735118,LastTimestamp:2026-10-29 22:41:52.55108063 +0000 UTC m=+0.364735118,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-80.ec2.internal,}" Oct 29 22:41:52.610714 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.610696 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vzsfk" Oct 29 22:41:52.633529 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.623000 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vzsfk" Oct 29 22:41:52.674566 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.674538 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 22:41:52.675719 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.675701 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 22:41:52.675798 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.675726 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 22:41:52.675798 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.675750 2581 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 22:41:52.675798 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:52.675792 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 29 22:41:52.687282 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.687266 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:41:52.690177 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.690161 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:41:52.690263 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.690189 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:41:52.690263 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.690201 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:41:52.690263 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.690223 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-80.ec2.internal" Oct 29 22:41:52.691254 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.691239 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:41:52.711568 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.711509 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-80.ec2.internal" Oct 29 22:41:52.711568 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:52.711530 2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-80.ec2.internal\": node \"ip-10-0-135-80.ec2.internal\" not found" Oct 29 22:41:52.759986 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:52.759956 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-80.ec2.internal\" not found" Oct 29 22:41:52.775890 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.775863 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal"] Oct 29 22:41:52.775984 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.775921 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:41:52.776948 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.776934 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:41:52.777038 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.776966 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:41:52.777038 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.776982 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:41:52.778217 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.778201 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:41:52.778817 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.778798 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Oct 29 22:41:52.778916 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.778830 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:41:52.778973 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.778942 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:41:52.778973 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.778969 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:41:52.779035 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.778979 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:41:52.779549 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.779536 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:41:52.779610 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.779560 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:41:52.779610 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.779594 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:41:52.780356 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.780340 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" Oct 29 22:41:52.780426 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.780365 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:41:52.781065 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.781049 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:41:52.781129 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.781078 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:41:52.781129 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.781092 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:41:52.798194 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:52.798169 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-80.ec2.internal\" not found" node="ip-10-0-135-80.ec2.internal" Oct 29 22:41:52.801829 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:52.801814 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-80.ec2.internal\" not found" node="ip-10-0-135-80.ec2.internal" Oct 29 22:41:52.835085 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.835062 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b94a7f102f3e9c0fab9ce63f581b4ba1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal\" (UID: \"b94a7f102f3e9c0fab9ce63f581b4ba1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Oct 29 22:41:52.835179 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.835086 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b94a7f102f3e9c0fab9ce63f581b4ba1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal\" (UID: \"b94a7f102f3e9c0fab9ce63f581b4ba1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Oct 29 22:41:52.835179 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.835103 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4239c6df062516f41cb9c5984a7633c8-config\") pod \"kube-apiserver-proxy-ip-10-0-135-80.ec2.internal\" (UID: \"4239c6df062516f41cb9c5984a7633c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" Oct 29 22:41:52.860788 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:52.860763 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-80.ec2.internal\" not found" Oct 29 22:41:52.935487 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.935454 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b94a7f102f3e9c0fab9ce63f581b4ba1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal\" (UID: \"b94a7f102f3e9c0fab9ce63f581b4ba1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Oct 29 22:41:52.935487 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.935490 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b94a7f102f3e9c0fab9ce63f581b4ba1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal\" (UID: \"b94a7f102f3e9c0fab9ce63f581b4ba1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Oct 29 22:41:52.935709 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.935512 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4239c6df062516f41cb9c5984a7633c8-config\") pod \"kube-apiserver-proxy-ip-10-0-135-80.ec2.internal\" (UID: \"4239c6df062516f41cb9c5984a7633c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" Oct 29 22:41:52.935709 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.935562 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4239c6df062516f41cb9c5984a7633c8-config\") pod \"kube-apiserver-proxy-ip-10-0-135-80.ec2.internal\" (UID: \"4239c6df062516f41cb9c5984a7633c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" Oct 29 22:41:52.935709 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.935563 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b94a7f102f3e9c0fab9ce63f581b4ba1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal\" (UID: \"b94a7f102f3e9c0fab9ce63f581b4ba1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Oct 29 22:41:52.935709 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:52.935563 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b94a7f102f3e9c0fab9ce63f581b4ba1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal\" (UID: \"b94a7f102f3e9c0fab9ce63f581b4ba1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Oct 29 22:41:52.961296 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:52.961272 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-80.ec2.internal\" not found" Oct 29 22:41:53.061859 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:53.061791 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-80.ec2.internal\" not found" Oct 29 22:41:53.101727 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:53.101708 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Oct 29 22:41:53.105547 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:53.105263 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" Oct 29 22:41:53.162229 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:53.162200 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-80.ec2.internal\" not found" Oct 29 22:41:53.262748 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:53.262719 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-80.ec2.internal\" not found" Oct 29 22:41:53.363279 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:53.363256 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-80.ec2.internal\" not found" Oct 29 22:41:54.538349 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:54.538317 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-80.ec2.internal\" not found" Oct 29 22:41:54.554474 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:54.554452 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 29 22:41:54.554620 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:54.554602 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 29 22:41:54.638864 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:54.638840 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-80.ec2.internal\" not found" Oct 29 22:41:54.676773 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:54.676740 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94a7f102f3e9c0fab9ce63f581b4ba1.slice/crio-72c20a30bd389edb03b22011f445c4d40c2fa5a93e32a280e4278313668c7aca WatchSource:0}: Error finding container 72c20a30bd389edb03b22011f445c4d40c2fa5a93e32a280e4278313668c7aca: Status 404 returned error can't find the container with id 72c20a30bd389edb03b22011f445c4d40c2fa5a93e32a280e4278313668c7aca Oct 29 22:41:54.677472 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:54.677450 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4239c6df062516f41cb9c5984a7633c8.slice/crio-901f5e90bef358345098700155850e5133969b5fd52e69a23d1452675c671c23 WatchSource:0}: Error finding container 901f5e90bef358345098700155850e5133969b5fd52e69a23d1452675c671c23: Status 404 returned error can't find the container with id 901f5e90bef358345098700155850e5133969b5fd52e69a23d1452675c671c23 Oct 29 22:41:54.681886 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:54.681873 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:41:54.702701 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:54.702639 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:36:53 +0000 UTC" deadline="2028-05-08 10:48:05.476893276 +0000 UTC" Oct 29 22:41:54.702701 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:54.702663 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13356h6m10.774232919s" Oct 29 22:41:54.737353 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:54.737330 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:41:54.753565 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:54.753516 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" event={"ID":"b94a7f102f3e9c0fab9ce63f581b4ba1","Type":"ContainerStarted","Data":"72c20a30bd389edb03b22011f445c4d40c2fa5a93e32a280e4278313668c7aca"} Oct 29 22:41:54.754517 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:54.754498 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" event={"ID":"4239c6df062516f41cb9c5984a7633c8","Type":"ContainerStarted","Data":"901f5e90bef358345098700155850e5133969b5fd52e69a23d1452675c671c23"} Oct 29 22:41:54.807673 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:54.807641 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Oct 29 22:41:54.863481 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:54.863453 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:41:54.865075 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:54.865060 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" Oct 29 22:41:54.909769 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:54.909743 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:41:54.975355 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:54.975280 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:41:55.068039 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.067998 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:41:55.406170 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.406086 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:41:55.587567 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.587538 2581 apiserver.go:52] "Watching apiserver" Oct 29 22:41:55.606149 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.606122 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 29 22:41:55.621751 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.621070 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 22:41:55.621751 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.621399 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-5czm4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal","openshift-multus/network-metrics-daemon-mxg8q","kube-system/konnectivity-agent-sc55n","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp","openshift-image-registry/node-ca-rwz4x","openshift-ovn-kubernetes/ovnkube-node-99ngw","kube-system/global-pull-secret-syncer-b2cf2","openshift-network-diagnostics/network-check-target-j8fft","openshift-cluster-node-tuning-operator/tuned-flzmd","openshift-multus/multus-additional-cni-plugins-vvp9h","openshift-multus/multus-qqk5c","openshift-network-operator/iptables-alerter-trj6w","kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal"] Oct 29 22:41:55.623129 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.623025 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.624074 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.624054 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.626501 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.626362 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.628130 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.627807 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.630390 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.630369 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-trj6w" Oct 29 22:41:55.633302 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.633285 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:41:55.633399 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:55.633368 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8fft" podUID="463bc340-b1dc-41e6-9203-fcd2672f8a17" Oct 29 22:41:55.635865 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.635846 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rwz4x" Oct 29 22:41:55.637511 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.637486 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.640166 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.638827 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5czm4" Oct 29 22:41:55.640403 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.640386 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sc55n" Oct 29 22:41:55.641036 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.641019 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xjmd7\"" Oct 29 22:41:55.641545 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.641506 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:41:55.643249 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.641598 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 22:41:55.643249 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.641670 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:41:55.643249 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:55.641725 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mxg8q" podUID="286133c3-abe5-4954-83a1-8665c1785b92" Oct 29 22:41:55.643249 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.642667 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:41:55.643249 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:55.642715 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2cf2" podUID="41ea529e-ab5d-419c-87bf-fe0bd1a6460f" Oct 29 22:41:55.643670 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.643654 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 22:41:55.710146 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.710120 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 22:41:55.718737 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.718707 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-run-systemd\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.718852 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.718748 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-run-k8s-cni-cncf-io\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.718852 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.718776 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-var-lib-cni-multus\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.718852 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.718797 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-dbus\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:41:55.718852 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.718817 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-cni-bin\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.718852 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.718832 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-run-multus-certs\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.719106 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.718854 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97fa3566-0c3f-4768-a437-3a89816303e0-tmp\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.719106 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.718878 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.719106 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.718898 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4525bd80-124f-4894-9fc0-b8235a0fe689-serviceca\") pod \"node-ca-rwz4x\" (UID: \"4525bd80-124f-4894-9fc0-b8235a0fe689\") " pod="openshift-image-registry/node-ca-rwz4x" Oct 29 22:41:55.719106 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.718912 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/97fa3566-0c3f-4768-a437-3a89816303e0-etc-tuned\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.719106 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.718926 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-var-lib-openvswitch\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.719106 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.718939 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4525bd80-124f-4894-9fc0-b8235a0fe689-host\") pod \"node-ca-rwz4x\" (UID: \"4525bd80-124f-4894-9fc0-b8235a0fe689\") " pod="openshift-image-registry/node-ca-rwz4x" Oct 29 22:41:55.719106 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.718970 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af926a18-1147-4786-8bd9-8a6727abb6fa-tmp-dir\") pod \"node-resolver-5czm4\" (UID: \"af926a18-1147-4786-8bd9-8a6727abb6fa\") " pod="openshift-dns/node-resolver-5czm4" Oct 29 22:41:55.719106 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.718990 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85325a00-d2ed-4c7d-b287-0fd5452f7e69-cni-binary-copy\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.719106 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719014 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-run-ovn\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.719106 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719031 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvrrh\" (UniqueName: \"kubernetes.io/projected/286133c3-abe5-4954-83a1-8665c1785b92-kube-api-access-qvrrh\") pod \"network-metrics-daemon-mxg8q\" (UID: \"286133c3-abe5-4954-83a1-8665c1785b92\") " pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:41:55.719106 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719059 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-run-netns\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.719106 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719076 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85sqm\" (UniqueName: \"kubernetes.io/projected/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-kube-api-access-85sqm\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.719106 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719100 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-systemd\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.719106 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719113 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-hostroot\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.719553 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719128 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-socket-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.719553 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719142 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.719553 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719157 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-host\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.719553 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719178 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71be8984-962e-4357-a596-b9d3931e3a75-ovnkube-script-lib\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.719553 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719207 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhmrs\" (UniqueName: \"kubernetes.io/projected/71be8984-962e-4357-a596-b9d3931e3a75-kube-api-access-rhmrs\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.719553 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719221 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-socket-dir-parent\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.719553 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719235 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-sys-fs\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.719553 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719248 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9-konnectivity-ca\") pod \"konnectivity-agent-sc55n\" (UID: \"2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9\") " pod="kube-system/konnectivity-agent-sc55n" Oct 29 22:41:55.719553 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719262 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plrh7\" (UniqueName: \"kubernetes.io/projected/97fa3566-0c3f-4768-a437-3a89816303e0-kube-api-access-plrh7\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.719553 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719276 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs\") pod \"network-metrics-daemon-mxg8q\" (UID: \"286133c3-abe5-4954-83a1-8665c1785b92\") " pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:41:55.719553 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719305 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-daemon-config\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.719553 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719324 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.719553 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719338 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-slash\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.719553 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719352 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-log-socket\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.719553 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719377 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z9zh\" (UniqueName: \"kubernetes.io/projected/af926a18-1147-4786-8bd9-8a6727abb6fa-kube-api-access-2z9zh\") pod \"node-resolver-5czm4\" (UID: \"af926a18-1147-4786-8bd9-8a6727abb6fa\") " pod="openshift-dns/node-resolver-5czm4" Oct 29 22:41:55.719553 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719392 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-kubelet-config\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:41:55.719964 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719406 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-sysctl-conf\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.719964 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719419 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-sys\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.719964 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719436 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-run-ovn-kubernetes\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.719964 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719450 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85325a00-d2ed-4c7d-b287-0fd5452f7e69-cnibin\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.719964 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719472 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85325a00-d2ed-4c7d-b287-0fd5452f7e69-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.719964 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719488 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-cni-binary-copy\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.719964 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719502 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-var-lib-cni-bin\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.719964 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719518 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-registration-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.719964 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719533 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdmbj\" (UniqueName: \"kubernetes.io/projected/4590699c-6eb3-4546-835f-924180048d8a-kube-api-access-jdmbj\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.719964 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719551 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-modprobe-d\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.719964 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719566 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrls7\" (UniqueName: \"kubernetes.io/projected/85325a00-d2ed-4c7d-b287-0fd5452f7e69-kube-api-access-vrls7\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.719964 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719585 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-conf-dir\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.719964 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719598 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-device-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.719964 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719645 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-sysconfig\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.719964 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719661 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-sysctl-d\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.719964 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719674 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-node-log\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.720512 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719694 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af926a18-1147-4786-8bd9-8a6727abb6fa-hosts-file\") pod \"node-resolver-5czm4\" (UID: \"af926a18-1147-4786-8bd9-8a6727abb6fa\") " pod="openshift-dns/node-resolver-5czm4" Oct 29 22:41:55.720512 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719711 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85325a00-d2ed-4c7d-b287-0fd5452f7e69-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.720512 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719733 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-cnibin\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.720512 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719749 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-systemd-units\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.720512 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719762 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-run-netns\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.720512 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719780 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-system-cni-dir\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.720512 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719793 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-etc-kubernetes\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.720512 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719806 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk8rz\" (UniqueName: \"kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz\") pod \"network-check-target-j8fft\" (UID: \"463bc340-b1dc-41e6-9203-fcd2672f8a17\") " pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:41:55.720512 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719821 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:41:55.720512 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719834 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-kubelet\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.720512 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719848 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-etc-openvswitch\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.720512 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719862 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656-iptables-alerter-script\") pod \"iptables-alerter-trj6w\" (UID: \"8d2cdca1-7303-4d6c-98ea-5eb6b7d16656\") " pod="openshift-network-operator/iptables-alerter-trj6w" Oct 29 22:41:55.720512 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719884 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9-agent-certs\") pod \"konnectivity-agent-sc55n\" (UID: \"2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9\") " pod="kube-system/konnectivity-agent-sc55n" Oct 29 22:41:55.720512 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719917 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-cni-netd\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.720512 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719935 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71be8984-962e-4357-a596-b9d3931e3a75-ovnkube-config\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.720512 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719959 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71be8984-962e-4357-a596-b9d3931e3a75-env-overrides\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.721296 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719984 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71be8984-962e-4357-a596-b9d3931e3a75-ovn-node-metrics-cert\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.721296 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.719998 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656-host-slash\") pod \"iptables-alerter-trj6w\" (UID: \"8d2cdca1-7303-4d6c-98ea-5eb6b7d16656\") " pod="openshift-network-operator/iptables-alerter-trj6w" Oct 29 22:41:55.721296 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.720012 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85325a00-d2ed-4c7d-b287-0fd5452f7e69-system-cni-dir\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.721296 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.720025 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-var-lib-kubelet\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.721296 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.720038 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hp7m\" (UniqueName: \"kubernetes.io/projected/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656-kube-api-access-7hp7m\") pod \"iptables-alerter-trj6w\" (UID: \"8d2cdca1-7303-4d6c-98ea-5eb6b7d16656\") " pod="openshift-network-operator/iptables-alerter-trj6w" Oct 29 22:41:55.721296 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.720057 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcb88\" (UniqueName: \"kubernetes.io/projected/4525bd80-124f-4894-9fc0-b8235a0fe689-kube-api-access-gcb88\") pod \"node-ca-rwz4x\" (UID: \"4525bd80-124f-4894-9fc0-b8235a0fe689\") " pod="openshift-image-registry/node-ca-rwz4x" Oct 29 22:41:55.721296 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.720079 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-lib-modules\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.721296 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.720100 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-run-openvswitch\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.721296 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.720124 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85325a00-d2ed-4c7d-b287-0fd5452f7e69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.721296 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.720138 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-cni-dir\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.721296 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.720151 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-kubernetes\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.721296 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.720164 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-run\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.721296 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.720177 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-var-lib-kubelet\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.739360 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.739329 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 22:41:55.739472 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.739406 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jv28c" Oct 29 22:41:55.740940 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.740914 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 22:41:55.741335 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.741309 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-28brx\"" Oct 29 22:41:55.741431 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.741419 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 22:41:55.741529 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.741504 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 22:41:55.741606 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.741539 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 22:41:55.741606 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.741580 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 22:41:55.741689 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.741601 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 22:41:55.741689 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.741667 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 22:41:55.741781 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.741698 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 22:41:55.741781 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.741770 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 22:41:55.741858 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.741835 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 22:41:55.741993 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.741978 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 22:41:55.742257 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.742051 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 22:41:55.742257 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.742093 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 22:41:55.742257 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.742151 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:41:55.742257 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.742180 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 22:41:55.742257 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.742203 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 22:41:55.742257 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.742236 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 22:41:55.742591 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.742241 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h2plp\"" Oct 29 22:41:55.746516 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.746352 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 22:41:55.746516 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.746359 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 22:41:55.746516 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.746378 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 22:41:55.746516 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.746349 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 22:41:55.746516 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.746448 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 22:41:55.746788 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.746562 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n7jjw\"" Oct 29 22:41:55.746788 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.746627 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rpfjl\"" Oct 29 22:41:55.746788 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.746704 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tj892\"" Oct 29 22:41:55.746788 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.746776 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-phcjr\"" Oct 29 22:41:55.746963 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.746836 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 22:41:55.746963 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.746933 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 22:41:55.748015 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.747297 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-fchfg\"" Oct 29 22:41:55.748015 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.747494 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m52c5\"" Oct 29 22:41:55.751532 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.751509 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jv28c" Oct 29 22:41:55.820713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.820685 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-hostroot\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.820713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.820715 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-socket-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.820929 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.820732 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.820929 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.820755 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-host\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.820929 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.820780 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71be8984-962e-4357-a596-b9d3931e3a75-ovnkube-script-lib\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.820929 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.820795 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-hostroot\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.820929 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.820805 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhmrs\" (UniqueName: \"kubernetes.io/projected/71be8984-962e-4357-a596-b9d3931e3a75-kube-api-access-rhmrs\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.820929 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.820841 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-socket-dir-parent\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.820929 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.820869 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-sys-fs\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.820929 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.820871 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-host\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.820929 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.820891 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-socket-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.820929 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.820890 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.820929 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.820894 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9-konnectivity-ca\") pod \"konnectivity-agent-sc55n\" (UID: \"2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9\") " pod="kube-system/konnectivity-agent-sc55n" Oct 29 22:41:55.821461 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.820951 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plrh7\" (UniqueName: \"kubernetes.io/projected/97fa3566-0c3f-4768-a437-3a89816303e0-kube-api-access-plrh7\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.821461 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.820959 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-socket-dir-parent\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.821461 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.820978 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs\") pod \"network-metrics-daemon-mxg8q\" (UID: \"286133c3-abe5-4954-83a1-8665c1785b92\") " pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:41:55.821461 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:55.821055 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:41:55.821461 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:55.821119 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs podName:286133c3-abe5-4954-83a1-8665c1785b92 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:56.321097877 +0000 UTC m=+3.060231971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs") pod "network-metrics-daemon-mxg8q" (UID: "286133c3-abe5-4954-83a1-8665c1785b92") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:41:55.821461 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821242 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-daemon-config\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.821461 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821273 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.821461 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821296 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-slash\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.821461 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821320 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-log-socket\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.821461 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821343 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z9zh\" (UniqueName: \"kubernetes.io/projected/af926a18-1147-4786-8bd9-8a6727abb6fa-kube-api-access-2z9zh\") pod \"node-resolver-5czm4\" (UID: \"af926a18-1147-4786-8bd9-8a6727abb6fa\") " pod="openshift-dns/node-resolver-5czm4" Oct 29 22:41:55.821461 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821371 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-kubelet-config\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:41:55.821461 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821393 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-sysctl-conf\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.821461 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821414 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-sys\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.821461 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821436 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-run-ovn-kubernetes\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.821461 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821460 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85325a00-d2ed-4c7d-b287-0fd5452f7e69-cnibin\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.822060 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821469 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9-konnectivity-ca\") pod \"konnectivity-agent-sc55n\" (UID: \"2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9\") " pod="kube-system/konnectivity-agent-sc55n" Oct 29 22:41:55.822060 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821485 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85325a00-d2ed-4c7d-b287-0fd5452f7e69-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.822060 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821492 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71be8984-962e-4357-a596-b9d3931e3a75-ovnkube-script-lib\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.822060 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821562 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-sys-fs\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.822060 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821609 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.822060 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821659 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-sys\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.822060 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821704 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-slash\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.822060 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821713 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-kubelet-config\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:41:55.822060 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821749 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-log-socket\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.822060 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821776 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-cni-binary-copy\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.822060 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821805 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-var-lib-cni-bin\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.822060 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821829 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-registration-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.822060 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821851 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-sysctl-conf\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.822060 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821853 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdmbj\" (UniqueName: \"kubernetes.io/projected/4590699c-6eb3-4546-835f-924180048d8a-kube-api-access-jdmbj\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.822060 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821890 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-modprobe-d\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.822060 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821918 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrls7\" (UniqueName: \"kubernetes.io/projected/85325a00-d2ed-4c7d-b287-0fd5452f7e69-kube-api-access-vrls7\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.822060 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821945 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-conf-dir\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.822713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821969 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-device-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.822713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821992 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-daemon-config\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.822713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.821993 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-sysconfig\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.822713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822031 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-sysconfig\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.822713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822034 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-sysctl-d\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.822713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822038 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85325a00-d2ed-4c7d-b287-0fd5452f7e69-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.822713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822061 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-node-log\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.822713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822090 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af926a18-1147-4786-8bd9-8a6727abb6fa-hosts-file\") pod \"node-resolver-5czm4\" (UID: \"af926a18-1147-4786-8bd9-8a6727abb6fa\") " pod="openshift-dns/node-resolver-5czm4" Oct 29 22:41:55.822713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822099 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-run-ovn-kubernetes\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.822713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822112 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85325a00-d2ed-4c7d-b287-0fd5452f7e69-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.822713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822135 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-cnibin\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.822713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822152 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-sysctl-d\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.822713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822158 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-systemd-units\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.822713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822148 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85325a00-d2ed-4c7d-b287-0fd5452f7e69-cnibin\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.822713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822203 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-run-netns\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.822713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822225 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-system-cni-dir\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.822713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822248 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-etc-kubernetes\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.823247 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822271 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk8rz\" (UniqueName: \"kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz\") pod \"network-check-target-j8fft\" (UID: \"463bc340-b1dc-41e6-9203-fcd2672f8a17\") " pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:41:55.823247 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822279 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-device-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.823247 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822284 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-modprobe-d\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.823247 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822292 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-var-lib-cni-bin\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.823247 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822296 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:41:55.823247 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822315 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-conf-dir\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.823247 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822333 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-kubelet\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.823247 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822339 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af926a18-1147-4786-8bd9-8a6727abb6fa-hosts-file\") pod \"node-resolver-5czm4\" (UID: \"af926a18-1147-4786-8bd9-8a6727abb6fa\") " pod="openshift-dns/node-resolver-5czm4" Oct 29 22:41:55.823247 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822359 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-cnibin\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.823247 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822364 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-node-log\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.823247 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822373 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-kubelet\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.823247 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822368 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-etc-openvswitch\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.823247 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822389 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-system-cni-dir\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.823247 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822418 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656-iptables-alerter-script\") pod \"iptables-alerter-trj6w\" (UID: \"8d2cdca1-7303-4d6c-98ea-5eb6b7d16656\") " pod="openshift-network-operator/iptables-alerter-trj6w" Oct 29 22:41:55.823247 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822423 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-etc-openvswitch\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.823247 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822432 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-registration-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.823247 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822444 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9-agent-certs\") pod \"konnectivity-agent-sc55n\" (UID: \"2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9\") " pod="kube-system/konnectivity-agent-sc55n" Oct 29 22:41:55.823247 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822466 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-systemd-units\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.823969 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822507 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-cni-netd\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.823969 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822511 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-run-netns\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.823969 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822539 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71be8984-962e-4357-a596-b9d3931e3a75-ovnkube-config\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.823969 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822546 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-cni-netd\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.823969 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:55.822586 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:41:55.823969 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822582 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-etc-kubernetes\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.823969 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822633 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71be8984-962e-4357-a596-b9d3931e3a75-env-overrides\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.823969 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:55.822678 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret podName:41ea529e-ab5d-419c-87bf-fe0bd1a6460f nodeName:}" failed. No retries permitted until 2026-10-29 22:41:56.322664279 +0000 UTC m=+3.061798369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret") pod "global-pull-secret-syncer-b2cf2" (UID: "41ea529e-ab5d-419c-87bf-fe0bd1a6460f") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:41:55.823969 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822721 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71be8984-962e-4357-a596-b9d3931e3a75-ovn-node-metrics-cert\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.823969 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822743 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656-host-slash\") pod \"iptables-alerter-trj6w\" (UID: \"8d2cdca1-7303-4d6c-98ea-5eb6b7d16656\") " pod="openshift-network-operator/iptables-alerter-trj6w" Oct 29 22:41:55.823969 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822759 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85325a00-d2ed-4c7d-b287-0fd5452f7e69-system-cni-dir\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.823969 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822774 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-var-lib-kubelet\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.823969 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822794 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hp7m\" (UniqueName: \"kubernetes.io/projected/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656-kube-api-access-7hp7m\") pod \"iptables-alerter-trj6w\" (UID: \"8d2cdca1-7303-4d6c-98ea-5eb6b7d16656\") " pod="openshift-network-operator/iptables-alerter-trj6w" Oct 29 22:41:55.823969 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822812 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656-host-slash\") pod \"iptables-alerter-trj6w\" (UID: \"8d2cdca1-7303-4d6c-98ea-5eb6b7d16656\") " pod="openshift-network-operator/iptables-alerter-trj6w" Oct 29 22:41:55.823969 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822818 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcb88\" (UniqueName: \"kubernetes.io/projected/4525bd80-124f-4894-9fc0-b8235a0fe689-kube-api-access-gcb88\") pod \"node-ca-rwz4x\" (UID: \"4525bd80-124f-4894-9fc0-b8235a0fe689\") " pod="openshift-image-registry/node-ca-rwz4x" Oct 29 22:41:55.823969 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822827 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85325a00-d2ed-4c7d-b287-0fd5452f7e69-system-cni-dir\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.823969 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822840 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-lib-modules\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.824478 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822864 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-run-openvswitch\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.824478 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822887 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85325a00-d2ed-4c7d-b287-0fd5452f7e69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.824478 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822910 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-cni-dir\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.824478 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822934 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-kubernetes\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.824478 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822956 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-run\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.824478 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822978 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-var-lib-kubelet\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.824478 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823002 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-run-systemd\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.824478 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.822998 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-run-openvswitch\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.824478 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823025 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-run-k8s-cni-cncf-io\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.824478 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823050 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-var-lib-cni-multus\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.824478 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823063 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-var-lib-kubelet\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.824478 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823077 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-dbus\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:41:55.824478 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823103 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-cni-bin\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.824478 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823124 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-run\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.824478 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823130 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-run-multus-certs\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.824478 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823156 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97fa3566-0c3f-4768-a437-3a89816303e0-tmp\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.824478 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823181 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-kubernetes\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.824478 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823207 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.825125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823235 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4525bd80-124f-4894-9fc0-b8235a0fe689-serviceca\") pod \"node-ca-rwz4x\" (UID: \"4525bd80-124f-4894-9fc0-b8235a0fe689\") " pod="openshift-image-registry/node-ca-rwz4x" Oct 29 22:41:55.825125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823260 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/97fa3566-0c3f-4768-a437-3a89816303e0-etc-tuned\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.825125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823292 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-var-lib-openvswitch\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.825125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823312 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4525bd80-124f-4894-9fc0-b8235a0fe689-host\") pod \"node-ca-rwz4x\" (UID: \"4525bd80-124f-4894-9fc0-b8235a0fe689\") " pod="openshift-image-registry/node-ca-rwz4x" Oct 29 22:41:55.825125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823328 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-cni-binary-copy\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.825125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823336 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af926a18-1147-4786-8bd9-8a6727abb6fa-tmp-dir\") pod \"node-resolver-5czm4\" (UID: \"af926a18-1147-4786-8bd9-8a6727abb6fa\") " pod="openshift-dns/node-resolver-5czm4" Oct 29 22:41:55.825125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823331 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85325a00-d2ed-4c7d-b287-0fd5452f7e69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.825125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823366 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85325a00-d2ed-4c7d-b287-0fd5452f7e69-cni-binary-copy\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.825125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823380 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-run-systemd\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.825125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823388 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-lib-modules\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.825125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823394 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-run-ovn\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.825125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823416 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-var-lib-cni-multus\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.825125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823423 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvrrh\" (UniqueName: \"kubernetes.io/projected/286133c3-abe5-4954-83a1-8665c1785b92-kube-api-access-qvrrh\") pod \"network-metrics-daemon-mxg8q\" (UID: \"286133c3-abe5-4954-83a1-8665c1785b92\") " pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:41:55.825125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823428 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-run-multus-certs\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.825125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823153 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-cni-dir\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.825125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823451 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-dbus\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:41:55.825125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823461 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-run-netns\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.825125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823473 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-run-k8s-cni-cncf-io\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.825719 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823488 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.825719 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823494 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4525bd80-124f-4894-9fc0-b8235a0fe689-host\") pod \"node-ca-rwz4x\" (UID: \"4525bd80-124f-4894-9fc0-b8235a0fe689\") " pod="openshift-image-registry/node-ca-rwz4x" Oct 29 22:41:55.825719 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823489 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85sqm\" (UniqueName: \"kubernetes.io/projected/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-kube-api-access-85sqm\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.825719 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823525 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85325a00-d2ed-4c7d-b287-0fd5452f7e69-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.825719 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823568 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-var-lib-kubelet\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.825719 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823608 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71be8984-962e-4357-a596-b9d3931e3a75-ovnkube-config\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.825719 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823625 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656-iptables-alerter-script\") pod \"iptables-alerter-trj6w\" (UID: \"8d2cdca1-7303-4d6c-98ea-5eb6b7d16656\") " pod="openshift-network-operator/iptables-alerter-trj6w" Oct 29 22:41:55.825719 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823646 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-var-lib-openvswitch\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.825719 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823651 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-cni-bin\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.825719 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823650 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-run-netns\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.825719 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823680 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-run-ovn\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.825719 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823739 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-systemd\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.825719 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823826 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-systemd\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.825719 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.823936 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af926a18-1147-4786-8bd9-8a6727abb6fa-tmp-dir\") pod \"node-resolver-5czm4\" (UID: \"af926a18-1147-4786-8bd9-8a6727abb6fa\") " pod="openshift-dns/node-resolver-5czm4" Oct 29 22:41:55.825719 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.824041 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85325a00-d2ed-4c7d-b287-0fd5452f7e69-cni-binary-copy\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.825719 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.824051 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4525bd80-124f-4894-9fc0-b8235a0fe689-serviceca\") pod \"node-ca-rwz4x\" (UID: \"4525bd80-124f-4894-9fc0-b8235a0fe689\") " pod="openshift-image-registry/node-ca-rwz4x" Oct 29 22:41:55.825719 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.824165 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 22:41:55.825719 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.824908 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71be8984-962e-4357-a596-b9d3931e3a75-env-overrides\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.828110 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.828070 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97fa3566-0c3f-4768-a437-3a89816303e0-tmp\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.828110 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.828094 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/97fa3566-0c3f-4768-a437-3a89816303e0-etc-tuned\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.828299 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.828178 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71be8984-962e-4357-a596-b9d3931e3a75-ovn-node-metrics-cert\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.828357 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.828335 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9-agent-certs\") pod \"konnectivity-agent-sc55n\" (UID: \"2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9\") " pod="kube-system/konnectivity-agent-sc55n" Oct 29 22:41:55.846204 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:55.846012 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:41:55.851394 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:55.846216 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:41:55.851394 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:55.846233 2581 projected.go:196] Error preparing data for projected volume kube-api-access-jk8rz for pod openshift-network-diagnostics/network-check-target-j8fft: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:41:55.851394 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:55.846292 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz podName:463bc340-b1dc-41e6-9203-fcd2672f8a17 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:56.346273279 +0000 UTC m=+3.085407569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jk8rz" (UniqueName: "kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz") pod "network-check-target-j8fft" (UID: "463bc340-b1dc-41e6-9203-fcd2672f8a17") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:41:55.851394 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.846872 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plrh7\" (UniqueName: \"kubernetes.io/projected/97fa3566-0c3f-4768-a437-3a89816303e0-kube-api-access-plrh7\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.851394 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.846958 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z9zh\" (UniqueName: \"kubernetes.io/projected/af926a18-1147-4786-8bd9-8a6727abb6fa-kube-api-access-2z9zh\") pod \"node-resolver-5czm4\" (UID: \"af926a18-1147-4786-8bd9-8a6727abb6fa\") " pod="openshift-dns/node-resolver-5czm4" Oct 29 22:41:55.851394 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.848226 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hp7m\" (UniqueName: \"kubernetes.io/projected/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656-kube-api-access-7hp7m\") pod \"iptables-alerter-trj6w\" (UID: \"8d2cdca1-7303-4d6c-98ea-5eb6b7d16656\") " pod="openshift-network-operator/iptables-alerter-trj6w" Oct 29 22:41:55.851394 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.848682 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrls7\" (UniqueName: \"kubernetes.io/projected/85325a00-d2ed-4c7d-b287-0fd5452f7e69-kube-api-access-vrls7\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.851394 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.848780 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcb88\" (UniqueName: \"kubernetes.io/projected/4525bd80-124f-4894-9fc0-b8235a0fe689-kube-api-access-gcb88\") pod \"node-ca-rwz4x\" (UID: \"4525bd80-124f-4894-9fc0-b8235a0fe689\") " pod="openshift-image-registry/node-ca-rwz4x" Oct 29 22:41:55.851394 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.849088 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdmbj\" (UniqueName: \"kubernetes.io/projected/4590699c-6eb3-4546-835f-924180048d8a-kube-api-access-jdmbj\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.851394 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.850100 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhmrs\" (UniqueName: \"kubernetes.io/projected/71be8984-962e-4357-a596-b9d3931e3a75-kube-api-access-rhmrs\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.851394 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.850318 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85sqm\" (UniqueName: \"kubernetes.io/projected/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-kube-api-access-85sqm\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.852054 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.852032 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvrrh\" (UniqueName: \"kubernetes.io/projected/286133c3-abe5-4954-83a1-8665c1785b92-kube-api-access-qvrrh\") pod \"network-metrics-daemon-mxg8q\" (UID: \"286133c3-abe5-4954-83a1-8665c1785b92\") " pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:41:55.936633 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.936602 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:41:55.951493 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.951472 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:41:55.956043 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.956020 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qqk5c" Oct 29 22:41:55.959539 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:55.959513 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85325a00_d2ed_4c7d_b287_0fd5452f7e69.slice/crio-f94f8986e84080f4850d25025683022e39863ad901a59bfeb9eb02e89d5837da WatchSource:0}: Error finding container f94f8986e84080f4850d25025683022e39863ad901a59bfeb9eb02e89d5837da: Status 404 returned error can't find the container with id f94f8986e84080f4850d25025683022e39863ad901a59bfeb9eb02e89d5837da Oct 29 22:41:55.965025 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:55.965002 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2863700d_26b0_4e9d_bd6a_ce8939cf3c9a.slice/crio-cea9c5c68a1c41e3617d64636095622a1097edea881f16f4fc74dadeea8e15e4 WatchSource:0}: Error finding container cea9c5c68a1c41e3617d64636095622a1097edea881f16f4fc74dadeea8e15e4: Status 404 returned error can't find the container with id cea9c5c68a1c41e3617d64636095622a1097edea881f16f4fc74dadeea8e15e4 Oct 29 22:41:55.973136 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.973119 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-trj6w" Oct 29 22:41:55.974268 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.974250 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:41:55.978470 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.978452 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rwz4x" Oct 29 22:41:55.982818 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:55.982794 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2cdca1_7303_4d6c_98ea_5eb6b7d16656.slice/crio-e3629e1ac73011582ae081f05d21007b7c932456f98e64abbab89ce4fc161673 WatchSource:0}: Error finding container e3629e1ac73011582ae081f05d21007b7c932456f98e64abbab89ce4fc161673: Status 404 returned error can't find the container with id e3629e1ac73011582ae081f05d21007b7c932456f98e64abbab89ce4fc161673 Oct 29 22:41:55.987474 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.987408 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:41:55.989202 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:55.989166 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71be8984_962e_4357_a596_b9d3931e3a75.slice/crio-5148e4d802089db3f519fa75c16f6d11b1e8d116ca1337134eb4edf466cbacb6 WatchSource:0}: Error finding container 5148e4d802089db3f519fa75c16f6d11b1e8d116ca1337134eb4edf466cbacb6: Status 404 returned error can't find the container with id 5148e4d802089db3f519fa75c16f6d11b1e8d116ca1337134eb4edf466cbacb6 Oct 29 22:41:55.993916 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.993894 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5czm4" Oct 29 22:41:55.999224 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:55.999161 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sc55n" Oct 29 22:41:56.012583 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:41:56.012552 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0ad66c_ca75_48ef_aa6b_89e0d2fb2db9.slice/crio-fd844cb3511c0f05c9f8c26c1c22d0494132ad245c74b1242938d0befbf75c70 WatchSource:0}: Error finding container fd844cb3511c0f05c9f8c26c1c22d0494132ad245c74b1242938d0befbf75c70: Status 404 returned error can't find the container with id fd844cb3511c0f05c9f8c26c1c22d0494132ad245c74b1242938d0befbf75c70 Oct 29 22:41:56.327174 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:56.327091 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs\") pod \"network-metrics-daemon-mxg8q\" (UID: \"286133c3-abe5-4954-83a1-8665c1785b92\") " pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:41:56.327174 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:56.327166 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:41:56.327398 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:56.327320 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:41:56.327398 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:56.327374 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret podName:41ea529e-ab5d-419c-87bf-fe0bd1a6460f nodeName:}" failed. No retries permitted until 2026-10-29 22:41:57.327358478 +0000 UTC m=+4.066492569 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret") pod "global-pull-secret-syncer-b2cf2" (UID: "41ea529e-ab5d-419c-87bf-fe0bd1a6460f") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:41:56.327713 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:56.327697 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:41:56.327790 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:56.327750 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs podName:286133c3-abe5-4954-83a1-8665c1785b92 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:57.327734182 +0000 UTC m=+4.066868274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs") pod "network-metrics-daemon-mxg8q" (UID: "286133c3-abe5-4954-83a1-8665c1785b92") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:41:56.427747 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:56.427705 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk8rz\" (UniqueName: \"kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz\") pod \"network-check-target-j8fft\" (UID: \"463bc340-b1dc-41e6-9203-fcd2672f8a17\") " pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:41:56.427914 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:56.427880 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:41:56.427914 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:56.427899 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:41:56.427914 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:56.427911 2581 projected.go:196] Error preparing data for projected volume kube-api-access-jk8rz for pod openshift-network-diagnostics/network-check-target-j8fft: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:41:56.428059 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:56.427972 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz podName:463bc340-b1dc-41e6-9203-fcd2672f8a17 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:57.427952847 +0000 UTC m=+4.167086925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jk8rz" (UniqueName: "kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz") pod "network-check-target-j8fft" (UID: "463bc340-b1dc-41e6-9203-fcd2672f8a17") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:41:56.753946 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:56.753891 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:36:55 +0000 UTC" deadline="2028-04-26 17:06:57.36818637 +0000 UTC" Oct 29 22:41:56.753946 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:56.753942 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13074h25m0.614246913s" Oct 29 22:41:56.779925 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:56.779892 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" event={"ID":"4239c6df062516f41cb9c5984a7633c8","Type":"ContainerStarted","Data":"71d1e51ebf55437eece7c4675c9d51959adc116f991ae36a042ba715a13e0355"} Oct 29 22:41:56.788321 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:56.788280 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sc55n" event={"ID":"2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9","Type":"ContainerStarted","Data":"fd844cb3511c0f05c9f8c26c1c22d0494132ad245c74b1242938d0befbf75c70"} Oct 29 22:41:56.795575 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:56.795543 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rwz4x" event={"ID":"4525bd80-124f-4894-9fc0-b8235a0fe689","Type":"ContainerStarted","Data":"82c4b5ea11fe463ac3c5b009a989c3434ee67cf2d2db2628c885902825057b10"} Oct 29 22:41:56.806486 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:56.806457 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-trj6w" event={"ID":"8d2cdca1-7303-4d6c-98ea-5eb6b7d16656","Type":"ContainerStarted","Data":"e3629e1ac73011582ae081f05d21007b7c932456f98e64abbab89ce4fc161673"} Oct 29 22:41:56.813404 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:56.813382 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vvp9h" event={"ID":"85325a00-d2ed-4c7d-b287-0fd5452f7e69","Type":"ContainerStarted","Data":"f94f8986e84080f4850d25025683022e39863ad901a59bfeb9eb02e89d5837da"} Oct 29 22:41:56.818911 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:56.818835 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-flzmd" event={"ID":"97fa3566-0c3f-4768-a437-3a89816303e0","Type":"ContainerStarted","Data":"8807c93dd434ca9ae035f9da4de5bce20dbab0d1617d810ddb499543344fce3e"} Oct 29 22:41:56.839275 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:56.838666 2581 generic.go:356] "Generic (PLEG): container finished" podID="b94a7f102f3e9c0fab9ce63f581b4ba1" containerID="feafcd201853adb3f768525d57f4e273d0fb25ac84315031e3df0a74c5419c57" exitCode=0 Oct 29 22:41:56.839275 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:56.838844 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" event={"ID":"b94a7f102f3e9c0fab9ce63f581b4ba1","Type":"ContainerDied","Data":"feafcd201853adb3f768525d57f4e273d0fb25ac84315031e3df0a74c5419c57"} Oct 29 22:41:56.853005 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:56.852931 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5czm4" event={"ID":"af926a18-1147-4786-8bd9-8a6727abb6fa","Type":"ContainerStarted","Data":"a289effe7cef40dca73995ac30e516b30174c991b7833f92e253ea116322a6d6"} Oct 29 22:41:56.872984 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:56.872909 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" event={"ID":"4590699c-6eb3-4546-835f-924180048d8a","Type":"ContainerStarted","Data":"028b9bc723b018c49696c401214b8ec85c9b1f9d54c7f29a10ddfb3600d43b32"} Oct 29 22:41:56.885410 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:56.884737 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" podStartSLOduration=2.884726821 podStartE2EDuration="2.884726821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:56.835257644 +0000 UTC m=+3.574391740" watchObservedRunningTime="2026-10-29 22:41:56.884726821 +0000 UTC m=+3.623860916" Oct 29 22:41:56.889687 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:56.889595 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" event={"ID":"71be8984-962e-4357-a596-b9d3931e3a75","Type":"ContainerStarted","Data":"5148e4d802089db3f519fa75c16f6d11b1e8d116ca1337134eb4edf466cbacb6"} Oct 29 22:41:56.906988 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:56.906959 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qqk5c" event={"ID":"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a","Type":"ContainerStarted","Data":"cea9c5c68a1c41e3617d64636095622a1097edea881f16f4fc74dadeea8e15e4"} Oct 29 22:41:57.340645 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:57.340557 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs\") pod \"network-metrics-daemon-mxg8q\" (UID: \"286133c3-abe5-4954-83a1-8665c1785b92\") " pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:41:57.340645 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:57.340633 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:41:57.340855 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:57.340769 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:41:57.340855 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:57.340831 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret podName:41ea529e-ab5d-419c-87bf-fe0bd1a6460f nodeName:}" failed. No retries permitted until 2026-10-29 22:41:59.34081351 +0000 UTC m=+6.079947599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret") pod "global-pull-secret-syncer-b2cf2" (UID: "41ea529e-ab5d-419c-87bf-fe0bd1a6460f") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:41:57.341488 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:57.341467 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:41:57.341578 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:57.341526 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs podName:286133c3-abe5-4954-83a1-8665c1785b92 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:59.341508727 +0000 UTC m=+6.080642825 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs") pod "network-metrics-daemon-mxg8q" (UID: "286133c3-abe5-4954-83a1-8665c1785b92") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:41:57.442387 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:57.441732 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk8rz\" (UniqueName: \"kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz\") pod \"network-check-target-j8fft\" (UID: \"463bc340-b1dc-41e6-9203-fcd2672f8a17\") " pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:41:57.442387 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:57.441945 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:41:57.442387 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:57.441963 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:41:57.442387 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:57.441975 2581 projected.go:196] Error preparing data for projected volume kube-api-access-jk8rz for pod openshift-network-diagnostics/network-check-target-j8fft: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:41:57.442387 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:57.442031 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz podName:463bc340-b1dc-41e6-9203-fcd2672f8a17 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:59.442015374 +0000 UTC m=+6.181149454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jk8rz" (UniqueName: "kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz") pod "network-check-target-j8fft" (UID: "463bc340-b1dc-41e6-9203-fcd2672f8a17") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:41:57.751377 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:57.751344 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:41:57.751566 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:57.751404 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:41:57.751566 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:57.751353 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:41:57.751566 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:57.751513 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8fft" podUID="463bc340-b1dc-41e6-9203-fcd2672f8a17" Oct 29 22:41:57.751716 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:57.751610 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mxg8q" podUID="286133c3-abe5-4954-83a1-8665c1785b92" Oct 29 22:41:57.751716 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:57.751690 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2cf2" podUID="41ea529e-ab5d-419c-87bf-fe0bd1a6460f" Oct 29 22:41:57.754945 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:57.754898 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:36:55 +0000 UTC" deadline="2028-08-08 04:56:32.839979754 +0000 UTC" Oct 29 22:41:57.754945 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:57.754943 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15558h14m35.0850404s" Oct 29 22:41:57.920281 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:57.920243 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" event={"ID":"b94a7f102f3e9c0fab9ce63f581b4ba1","Type":"ContainerStarted","Data":"a2ccb0138cf51229483fc46416519e08e032de50f25307865d6e63a6a4cc0fd1"} Oct 29 22:41:57.965772 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:57.965708 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" podStartSLOduration=3.9656895739999998 podStartE2EDuration="3.965689574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:57.96520517 +0000 UTC m=+4.704339465" watchObservedRunningTime="2026-10-29 22:41:57.965689574 +0000 UTC m=+4.704823670" Oct 29 22:41:58.927494 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:58.926658 2581 generic.go:356] "Generic (PLEG): container finished" podID="85325a00-d2ed-4c7d-b287-0fd5452f7e69" containerID="e96d06cb90b7322155ae1238e68bd7752d79d6ac6863c4a4a461dfb04c204660" exitCode=0 Oct 29 22:41:58.927494 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:58.927308 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vvp9h" event={"ID":"85325a00-d2ed-4c7d-b287-0fd5452f7e69","Type":"ContainerDied","Data":"e96d06cb90b7322155ae1238e68bd7752d79d6ac6863c4a4a461dfb04c204660"} Oct 29 22:41:59.358395 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:59.358317 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs\") pod \"network-metrics-daemon-mxg8q\" (UID: \"286133c3-abe5-4954-83a1-8665c1785b92\") " pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:41:59.358563 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:59.358406 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:41:59.358563 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:59.358552 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:41:59.358673 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:59.358612 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret podName:41ea529e-ab5d-419c-87bf-fe0bd1a6460f nodeName:}" failed. No retries permitted until 2026-10-29 22:42:03.358594739 +0000 UTC m=+10.097728827 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret") pod "global-pull-secret-syncer-b2cf2" (UID: "41ea529e-ab5d-419c-87bf-fe0bd1a6460f") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:41:59.359052 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:59.359032 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:41:59.359157 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:59.359086 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs podName:286133c3-abe5-4954-83a1-8665c1785b92 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:03.359071511 +0000 UTC m=+10.098205585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs") pod "network-metrics-daemon-mxg8q" (UID: "286133c3-abe5-4954-83a1-8665c1785b92") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:41:59.459772 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:59.459737 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk8rz\" (UniqueName: \"kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz\") pod \"network-check-target-j8fft\" (UID: \"463bc340-b1dc-41e6-9203-fcd2672f8a17\") " pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:41:59.459941 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:59.459924 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:41:59.459987 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:59.459954 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:41:59.459987 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:59.459976 2581 projected.go:196] Error preparing data for projected volume kube-api-access-jk8rz for pod openshift-network-diagnostics/network-check-target-j8fft: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:41:59.460059 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:59.460045 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz podName:463bc340-b1dc-41e6-9203-fcd2672f8a17 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:03.460017151 +0000 UTC m=+10.199151243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jk8rz" (UniqueName: "kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz") pod "network-check-target-j8fft" (UID: "463bc340-b1dc-41e6-9203-fcd2672f8a17") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:41:59.750551 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:59.750513 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:41:59.750738 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:59.750669 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2cf2" podUID="41ea529e-ab5d-419c-87bf-fe0bd1a6460f" Oct 29 22:41:59.751050 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:59.751027 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:41:59.751167 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:59.751147 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mxg8q" podUID="286133c3-abe5-4954-83a1-8665c1785b92" Oct 29 22:41:59.751249 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:41:59.751210 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:41:59.751291 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:41:59.751275 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8fft" podUID="463bc340-b1dc-41e6-9203-fcd2672f8a17" Oct 29 22:42:00.939851 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:00.939024 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sc55n" event={"ID":"2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9","Type":"ContainerStarted","Data":"fbad698a5c18150d1f48e2453d8ad30e49a168eb33902d019441810b00afc1ee"} Oct 29 22:42:00.948172 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:00.947461 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rwz4x" event={"ID":"4525bd80-124f-4894-9fc0-b8235a0fe689","Type":"ContainerStarted","Data":"89f90c8aed08094baebb693120b9f263f9a1e3703a851f5249e6958d5c18e4ab"} Oct 29 22:42:01.037523 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:01.037474 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sc55n" podStartSLOduration=3.055632988 podStartE2EDuration="7.037457965s" totalImagesPullingTime="3.981824977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:00.985371422 +0000 UTC m=+7.724505519" watchObservedRunningTime="2026-10-29 22:42:01.037457965 +0000 UTC m=+7.776592060" Oct 29 22:42:01.254422 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:01.254306 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sc55n" Oct 29 22:42:01.255167 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:01.255144 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sc55n" Oct 29 22:42:01.301288 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:01.301156 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rwz4x" podStartSLOduration=2.614557599 podStartE2EDuration="7.301135733s" totalImagesPullingTime="4.686578134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:01.038263847 +0000 UTC m=+7.777397955" watchObservedRunningTime="2026-10-29 22:42:01.301135733 +0000 UTC m=+8.040269832" Oct 29 22:42:01.755409 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:01.755373 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:01.755409 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:01.755408 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:01.755660 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:01.755506 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2cf2" podUID="41ea529e-ab5d-419c-87bf-fe0bd1a6460f" Oct 29 22:42:01.755660 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:01.755589 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mxg8q" podUID="286133c3-abe5-4954-83a1-8665c1785b92" Oct 29 22:42:01.755660 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:01.755625 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:01.755843 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:01.755700 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8fft" podUID="463bc340-b1dc-41e6-9203-fcd2672f8a17" Oct 29 22:42:01.952626 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:01.952588 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" event={"ID":"4590699c-6eb3-4546-835f-924180048d8a","Type":"ContainerStarted","Data":"60dd883e4c3a224d37f5b1548fc334ce0e8eaa410ec01aeb6576244a0b61362d"} Oct 29 22:42:02.955079 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:02.954574 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:42:03.200869 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:03.200836 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 22:42:03.395746 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:03.395710 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs\") pod \"network-metrics-daemon-mxg8q\" (UID: \"286133c3-abe5-4954-83a1-8665c1785b92\") " pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:03.395884 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:03.395770 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:03.395884 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:03.395881 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:42:03.395976 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:03.395966 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret podName:41ea529e-ab5d-419c-87bf-fe0bd1a6460f nodeName:}" failed. No retries permitted until 2026-10-29 22:42:11.395952393 +0000 UTC m=+18.135086481 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret") pod "global-pull-secret-syncer-b2cf2" (UID: "41ea529e-ab5d-419c-87bf-fe0bd1a6460f") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:42:03.396284 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:03.396270 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:42:03.396334 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:03.396315 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs podName:286133c3-abe5-4954-83a1-8665c1785b92 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:11.396304051 +0000 UTC m=+18.135438138 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs") pod "network-metrics-daemon-mxg8q" (UID: "286133c3-abe5-4954-83a1-8665c1785b92") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:42:03.496694 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:03.496075 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk8rz\" (UniqueName: \"kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz\") pod \"network-check-target-j8fft\" (UID: \"463bc340-b1dc-41e6-9203-fcd2672f8a17\") " pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:03.496694 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:03.496275 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:42:03.496694 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:03.496297 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:42:03.496694 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:03.496310 2581 projected.go:196] Error preparing data for projected volume kube-api-access-jk8rz for pod openshift-network-diagnostics/network-check-target-j8fft: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:42:03.496694 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:03.496365 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz podName:463bc340-b1dc-41e6-9203-fcd2672f8a17 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:11.496347399 +0000 UTC m=+18.235481474 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jk8rz" (UniqueName: "kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz") pod "network-check-target-j8fft" (UID: "463bc340-b1dc-41e6-9203-fcd2672f8a17") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:42:03.665461 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:03.665364 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:42:03.200859924Z","UUID":"606cfb13-c6a6-4e9b-a116-bae9f09c97db","Handler":null,"Name":"","Endpoint":""} Oct 29 22:42:03.666929 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:03.666903 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 22:42:03.667049 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:03.666939 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 22:42:03.754995 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:03.754880 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:03.755158 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:03.755013 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2cf2" podUID="41ea529e-ab5d-419c-87bf-fe0bd1a6460f" Oct 29 22:42:03.755418 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:03.755398 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:03.755528 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:03.755505 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mxg8q" podUID="286133c3-abe5-4954-83a1-8665c1785b92" Oct 29 22:42:03.755589 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:03.755575 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:03.755670 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:03.755643 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8fft" podUID="463bc340-b1dc-41e6-9203-fcd2672f8a17" Oct 29 22:42:03.960596 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:03.960555 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" event={"ID":"4590699c-6eb3-4546-835f-924180048d8a","Type":"ContainerStarted","Data":"68366a1d6dde46e3a4c15c3eb67788e55e254cf44370062842a9ffefc1785b95"} Oct 29 22:42:03.962128 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:03.962101 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qqk5c" event={"ID":"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a","Type":"ContainerStarted","Data":"538243a1e4c68485b98725cbefc813eab54ff755160f196e172ee432e580bbfc"} Oct 29 22:42:03.966476 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:03.966449 2581 generic.go:356] "Generic (PLEG): container finished" podID="85325a00-d2ed-4c7d-b287-0fd5452f7e69" containerID="7912f888f1d0da0c4e179a31895cf9c9bacaa8d47d432fb4131bc6e2746498d2" exitCode=0 Oct 29 22:42:03.966583 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:03.966492 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vvp9h" event={"ID":"85325a00-d2ed-4c7d-b287-0fd5452f7e69","Type":"ContainerDied","Data":"7912f888f1d0da0c4e179a31895cf9c9bacaa8d47d432fb4131bc6e2746498d2"} Oct 29 22:42:04.014988 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:04.014887 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-qqk5c" podStartSLOduration=3.866029293 podStartE2EDuration="11.014869281s" totalImagesPullingTime="7.148839988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:04.014608096 +0000 UTC m=+10.753742192" watchObservedRunningTime="2026-10-29 22:42:04.014869281 +0000 UTC m=+10.754003379" Oct 29 22:42:04.970717 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:04.970322 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5czm4" event={"ID":"af926a18-1147-4786-8bd9-8a6727abb6fa","Type":"ContainerStarted","Data":"ff00c9703ee0f818353795b7a05e1a4052bff774d8f03e166b4cd3e8706e9396"} Oct 29 22:42:04.973083 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:04.973049 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" event={"ID":"4590699c-6eb3-4546-835f-924180048d8a","Type":"ContainerStarted","Data":"d5316ac02e6cac851d7a00ac09abf1b63d40c69488a769611e32a58ac10a2e7d"} Oct 29 22:42:04.974951 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:04.974922 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-trj6w" event={"ID":"8d2cdca1-7303-4d6c-98ea-5eb6b7d16656","Type":"ContainerStarted","Data":"ec6c128fd033f30c7d9cd5dc65a98b80e91389caa67020d227547141e7fd3d31"} Oct 29 22:42:04.978479 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:04.978453 2581 generic.go:356] "Generic (PLEG): container finished" podID="85325a00-d2ed-4c7d-b287-0fd5452f7e69" containerID="28dc5d6af575caecd2f55645a5d1a9593e92ab730789225034ce54f0eaa2cc56" exitCode=0 Oct 29 22:42:04.978590 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:04.978531 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vvp9h" event={"ID":"85325a00-d2ed-4c7d-b287-0fd5452f7e69","Type":"ContainerDied","Data":"28dc5d6af575caecd2f55645a5d1a9593e92ab730789225034ce54f0eaa2cc56"} Oct 29 22:42:05.066462 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:05.066401 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5czm4" podStartSLOduration=3.23993326 podStartE2EDuration="11.066381619s" totalImagesPullingTime="7.826448359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:05.015379738 +0000 UTC m=+11.754513838" watchObservedRunningTime="2026-10-29 22:42:05.066381619 +0000 UTC m=+11.805515718" Oct 29 22:42:05.113159 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:05.113109 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-trj6w" podStartSLOduration=4.590483096 podStartE2EDuration="12.113089942s" totalImagesPullingTime="7.522606846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:05.112222932 +0000 UTC m=+11.851357028" watchObservedRunningTime="2026-10-29 22:42:05.113089942 +0000 UTC m=+11.852224040" Oct 29 22:42:05.160045 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:05.159987 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" podStartSLOduration=3.466134279 podStartE2EDuration="11.15997217s" totalImagesPullingTime="7.693837891s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:05.159959409 +0000 UTC m=+11.899093505" watchObservedRunningTime="2026-10-29 22:42:05.15997217 +0000 UTC m=+11.899106248" Oct 29 22:42:05.166064 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:05.166031 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sc55n" Oct 29 22:42:05.166317 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:05.166215 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:42:05.166798 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:05.166757 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sc55n" Oct 29 22:42:05.752427 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:05.750832 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:05.752427 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:05.750855 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:05.752427 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:05.751032 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2cf2" podUID="41ea529e-ab5d-419c-87bf-fe0bd1a6460f" Oct 29 22:42:05.752427 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:05.751072 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:05.752427 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:05.751215 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8fft" podUID="463bc340-b1dc-41e6-9203-fcd2672f8a17" Oct 29 22:42:05.752427 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:05.751305 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mxg8q" podUID="286133c3-abe5-4954-83a1-8665c1785b92" Oct 29 22:42:05.984056 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:05.984025 2581 generic.go:356] "Generic (PLEG): container finished" podID="85325a00-d2ed-4c7d-b287-0fd5452f7e69" containerID="7525ecf3c4fb8cd0826fe365d3519374edf095e2c3c378cc5a1c5939168df5a5" exitCode=0 Oct 29 22:42:05.984513 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:05.984146 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vvp9h" event={"ID":"85325a00-d2ed-4c7d-b287-0fd5452f7e69","Type":"ContainerDied","Data":"7525ecf3c4fb8cd0826fe365d3519374edf095e2c3c378cc5a1c5939168df5a5"} Oct 29 22:42:06.988646 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:06.988423 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-flzmd" event={"ID":"97fa3566-0c3f-4768-a437-3a89816303e0","Type":"ContainerStarted","Data":"72360af30010942b0c0456747fc25c677e7d5478030ac54d6def9e8c9ec3ad96"} Oct 29 22:42:07.754141 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:07.754114 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:07.754281 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:07.754256 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2cf2" podUID="41ea529e-ab5d-419c-87bf-fe0bd1a6460f" Oct 29 22:42:07.754371 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:07.754330 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:07.754428 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:07.754411 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mxg8q" podUID="286133c3-abe5-4954-83a1-8665c1785b92" Oct 29 22:42:07.754475 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:07.754461 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:07.754557 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:07.754539 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8fft" podUID="463bc340-b1dc-41e6-9203-fcd2672f8a17" Oct 29 22:42:07.994132 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:07.994091 2581 generic.go:356] "Generic (PLEG): container finished" podID="85325a00-d2ed-4c7d-b287-0fd5452f7e69" containerID="8966cb9af80afb5802f7a7ecf814af8a3c9943662aca26aa159e285ae3e94f82" exitCode=0 Oct 29 22:42:07.994880 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:07.994180 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vvp9h" event={"ID":"85325a00-d2ed-4c7d-b287-0fd5452f7e69","Type":"ContainerDied","Data":"8966cb9af80afb5802f7a7ecf814af8a3c9943662aca26aa159e285ae3e94f82"} Oct 29 22:42:08.077003 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:08.076956 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-flzmd" podStartSLOduration=5.082574278 podStartE2EDuration="15.076942481s" totalImagesPullingTime="9.994368203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:07.036990846 +0000 UTC m=+13.776124942" watchObservedRunningTime="2026-10-29 22:42:08.076942481 +0000 UTC m=+14.816076576" Oct 29 22:42:08.999437 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:08.999400 2581 generic.go:356] "Generic (PLEG): container finished" podID="85325a00-d2ed-4c7d-b287-0fd5452f7e69" containerID="6240c2e598a9c0ce9990608cd183149def9de5756569d7b596be7796c6d9a55e" exitCode=0 Oct 29 22:42:08.999437 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:08.999442 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vvp9h" event={"ID":"85325a00-d2ed-4c7d-b287-0fd5452f7e69","Type":"ContainerDied","Data":"6240c2e598a9c0ce9990608cd183149def9de5756569d7b596be7796c6d9a55e"} Oct 29 22:42:09.754461 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:09.754258 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:09.754609 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:09.754265 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:09.754609 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:09.754539 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8fft" podUID="463bc340-b1dc-41e6-9203-fcd2672f8a17" Oct 29 22:42:09.754609 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:09.754312 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:09.754737 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:09.754605 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2cf2" podUID="41ea529e-ab5d-419c-87bf-fe0bd1a6460f" Oct 29 22:42:09.754737 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:09.754670 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mxg8q" podUID="286133c3-abe5-4954-83a1-8665c1785b92" Oct 29 22:42:10.005126 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:10.005042 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vvp9h" event={"ID":"85325a00-d2ed-4c7d-b287-0fd5452f7e69","Type":"ContainerStarted","Data":"7983b045ab6b09340291c76b3b549f119fb36c638dbed77d0c39654aedf792c7"} Oct 29 22:42:10.069943 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:10.069895 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vvp9h" podStartSLOduration=7.151732342 podStartE2EDuration="17.069881142s" totalImagesPullingTime="9.9181488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:10.069480936 +0000 UTC m=+16.808615033" watchObservedRunningTime="2026-10-29 22:42:10.069881142 +0000 UTC m=+16.809015237" Oct 29 22:42:11.011646 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:11.011610 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" event={"ID":"71be8984-962e-4357-a596-b9d3931e3a75","Type":"ContainerStarted","Data":"50108fce07033eef8560731bac7293a7369c0ecd9b9d88f6e743cb82d890e82b"} Oct 29 22:42:11.012118 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:11.011681 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" event={"ID":"71be8984-962e-4357-a596-b9d3931e3a75","Type":"ContainerStarted","Data":"a473427b92a3cff5c5db814a8e3e6cfedc46f7d1ec921bd74f5b2a3c0e930b6b"} Oct 29 22:42:11.012118 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:11.011713 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" event={"ID":"71be8984-962e-4357-a596-b9d3931e3a75","Type":"ContainerStarted","Data":"74e1e13ee732f444b9b99ddd441d54c9be69153f4a448171a58053fb52e4d39b"} Oct 29 22:42:11.012118 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:11.011742 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" event={"ID":"71be8984-962e-4357-a596-b9d3931e3a75","Type":"ContainerStarted","Data":"97a769f0801a3b08d20a767a2677902e8ed44419d1c90ee6bf97e4f371151713"} Oct 29 22:42:11.012118 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:11.011775 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" event={"ID":"71be8984-962e-4357-a596-b9d3931e3a75","Type":"ContainerStarted","Data":"cb53aa9edd21de09aaef774c4350a6f934ad13e121650a90a07c6a3c768340c9"} Oct 29 22:42:11.012118 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:11.011803 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" event={"ID":"71be8984-962e-4357-a596-b9d3931e3a75","Type":"ContainerStarted","Data":"3bc41e93f5a93614e9edacbe7772c5f9fec2fd30dcbfff8a505dd0838ea52999"} Oct 29 22:42:11.450682 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:11.450645 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs\") pod \"network-metrics-daemon-mxg8q\" (UID: \"286133c3-abe5-4954-83a1-8665c1785b92\") " pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:11.450865 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:11.450720 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:11.450865 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:11.450791 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:42:11.450865 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:11.450826 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:42:11.450865 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:11.450852 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs podName:286133c3-abe5-4954-83a1-8665c1785b92 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:27.450838189 +0000 UTC m=+34.189972263 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs") pod "network-metrics-daemon-mxg8q" (UID: "286133c3-abe5-4954-83a1-8665c1785b92") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:42:11.451007 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:11.450874 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret podName:41ea529e-ab5d-419c-87bf-fe0bd1a6460f nodeName:}" failed. No retries permitted until 2026-10-29 22:42:27.450859818 +0000 UTC m=+34.189993909 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret") pod "global-pull-secret-syncer-b2cf2" (UID: "41ea529e-ab5d-419c-87bf-fe0bd1a6460f") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:42:11.551104 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:11.551064 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk8rz\" (UniqueName: \"kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz\") pod \"network-check-target-j8fft\" (UID: \"463bc340-b1dc-41e6-9203-fcd2672f8a17\") " pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:11.551277 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:11.551243 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:42:11.551277 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:11.551269 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:42:11.551348 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:11.551280 2581 projected.go:196] Error preparing data for projected volume kube-api-access-jk8rz for pod openshift-network-diagnostics/network-check-target-j8fft: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:42:11.551348 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:11.551336 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz podName:463bc340-b1dc-41e6-9203-fcd2672f8a17 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:27.551316915 +0000 UTC m=+34.290451003 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jk8rz" (UniqueName: "kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz") pod "network-check-target-j8fft" (UID: "463bc340-b1dc-41e6-9203-fcd2672f8a17") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:42:11.751147 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:11.751060 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:11.751147 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:11.751088 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:11.751364 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:11.751065 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:11.751364 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:11.751220 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mxg8q" podUID="286133c3-abe5-4954-83a1-8665c1785b92" Oct 29 22:42:11.751364 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:11.751299 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8fft" podUID="463bc340-b1dc-41e6-9203-fcd2672f8a17" Oct 29 22:42:11.751364 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:11.751342 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2cf2" podUID="41ea529e-ab5d-419c-87bf-fe0bd1a6460f" Oct 29 22:42:13.019759 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:13.019558 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" event={"ID":"71be8984-962e-4357-a596-b9d3931e3a75","Type":"ContainerStarted","Data":"51b0f4b19a2aa0d940897c4dc6b6211ef0f03e6dd9f4ac2d9a51224a9f61823c"} Oct 29 22:42:13.753842 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:13.753812 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:13.754047 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:13.753882 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:13.754047 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:13.753905 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:13.754047 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:13.754001 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8fft" podUID="463bc340-b1dc-41e6-9203-fcd2672f8a17" Oct 29 22:42:13.754047 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:13.754030 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mxg8q" podUID="286133c3-abe5-4954-83a1-8665c1785b92" Oct 29 22:42:13.754284 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:13.754095 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2cf2" podUID="41ea529e-ab5d-419c-87bf-fe0bd1a6460f" Oct 29 22:42:15.027006 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:15.026966 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" event={"ID":"71be8984-962e-4357-a596-b9d3931e3a75","Type":"ContainerStarted","Data":"25eadda9f9c56b8aa9893985c3115ed86cd6808196154a41a00a2d22901843f9"} Oct 29 22:42:15.027813 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:15.027246 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:15.027813 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:15.027305 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:15.027813 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:15.027352 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:15.044447 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:15.044422 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:15.044581 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:15.044533 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:15.129324 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:15.129273 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" podStartSLOduration=8.166818826 podStartE2EDuration="22.129260733s" totalImagesPullingTime="13.962441907s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:15.128962866 +0000 UTC m=+21.868096962" watchObservedRunningTime="2026-10-29 22:42:15.129260733 +0000 UTC m=+21.868394828" Oct 29 22:42:15.750703 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:15.750672 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:15.750703 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:15.750698 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:15.750887 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:15.750803 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8fft" podUID="463bc340-b1dc-41e6-9203-fcd2672f8a17" Oct 29 22:42:15.750887 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:15.750850 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:15.750963 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:15.750942 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mxg8q" podUID="286133c3-abe5-4954-83a1-8665c1785b92" Oct 29 22:42:15.751024 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:15.751010 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2cf2" podUID="41ea529e-ab5d-419c-87bf-fe0bd1a6460f" Oct 29 22:42:16.555398 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:16.555364 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j8fft"] Oct 29 22:42:16.555833 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:16.555495 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:16.555833 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:16.555603 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8fft" podUID="463bc340-b1dc-41e6-9203-fcd2672f8a17" Oct 29 22:42:16.557838 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:16.557811 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mxg8q"] Oct 29 22:42:16.557933 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:16.557916 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:16.558031 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:16.558014 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mxg8q" podUID="286133c3-abe5-4954-83a1-8665c1785b92" Oct 29 22:42:16.569741 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:16.569715 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b2cf2"] Oct 29 22:42:16.569840 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:16.569800 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:16.569897 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:16.569882 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2cf2" podUID="41ea529e-ab5d-419c-87bf-fe0bd1a6460f" Oct 29 22:42:17.751491 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:17.751462 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:17.751948 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:17.751572 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mxg8q" podUID="286133c3-abe5-4954-83a1-8665c1785b92" Oct 29 22:42:18.751721 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:18.751488 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:18.752146 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:18.751522 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:18.752146 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:18.751781 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8fft" podUID="463bc340-b1dc-41e6-9203-fcd2672f8a17" Oct 29 22:42:18.752146 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:18.751816 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2cf2" podUID="41ea529e-ab5d-419c-87bf-fe0bd1a6460f" Oct 29 22:42:19.750906 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:19.750864 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:19.751101 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:19.751008 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mxg8q" podUID="286133c3-abe5-4954-83a1-8665c1785b92" Oct 29 22:42:20.751582 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:20.751490 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:20.751582 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:20.751518 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:20.752022 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:20.751610 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2cf2" podUID="41ea529e-ab5d-419c-87bf-fe0bd1a6460f" Oct 29 22:42:20.752022 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:20.751725 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8fft" podUID="463bc340-b1dc-41e6-9203-fcd2672f8a17" Oct 29 22:42:21.531433 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.531411 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeReady" Oct 29 22:42:21.531547 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.531524 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 29 22:42:21.585316 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.585287 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7hz4v"] Oct 29 22:42:21.588077 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.588052 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7hz4v" Oct 29 22:42:21.588536 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.588517 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fh7rx"] Oct 29 22:42:21.591398 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.591380 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fh7rx" Oct 29 22:42:21.594628 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.594608 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k8xnq"] Oct 29 22:42:21.597227 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.597210 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh"] Oct 29 22:42:21.597304 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.597259 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-k8xnq" Oct 29 22:42:21.599766 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.599744 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" Oct 29 22:42:21.600749 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.600059 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-868dt"] Oct 29 22:42:21.603477 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.603457 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cf56cb474-gx24x"] Oct 29 22:42:21.605420 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.605380 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-868dt" Oct 29 22:42:21.608513 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.608495 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh"] Oct 29 22:42:21.608635 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.608617 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.611092 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.611070 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7945df5b5-pcgp6"] Oct 29 22:42:21.611179 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.611162 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:21.614233 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.614215 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-rwfmh"] Oct 29 22:42:21.615760 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.615398 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:21.615760 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.615445 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 29 22:42:21.615760 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.615448 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 29 22:42:21.618319 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.618302 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-9qhv9"] Oct 29 22:42:21.618466 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.618422 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-rwfmh" Oct 29 22:42:21.620797 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.620760 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-cz727\"" Oct 29 22:42:21.621263 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.621243 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b"] Oct 29 22:42:21.621379 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.621360 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:21.624517 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.624343 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf"] Oct 29 22:42:21.624517 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.624443 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:21.627342 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.627321 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-lm55w"] Oct 29 22:42:21.627448 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.627432 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" Oct 29 22:42:21.630211 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.630181 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-kw5c9"] Oct 29 22:42:21.630294 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.630257 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:21.633099 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.633080 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78f4dff4c9-87jff"] Oct 29 22:42:21.633264 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.633230 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kw5c9" Oct 29 22:42:21.635914 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.635895 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7hz4v"] Oct 29 22:42:21.635995 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.635934 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fh7rx"] Oct 29 22:42:21.635995 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.635954 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k8xnq"] Oct 29 22:42:21.635995 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.635967 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gj6vs"] Oct 29 22:42:21.636137 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.636048 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.639075 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.639057 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qbv9d"] Oct 29 22:42:21.639210 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.639176 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:21.641735 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.641716 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qbv9d" Oct 29 22:42:21.648636 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.648617 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh"] Oct 29 22:42:21.651249 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.651228 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh"] Oct 29 22:42:21.652445 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.652422 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cf56cb474-gx24x"] Oct 29 22:42:21.659555 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.659534 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-9qhv9"] Oct 29 22:42:21.660238 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.660221 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-rwfmh"] Oct 29 22:42:21.661157 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.661135 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf"] Oct 29 22:42:21.663394 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.663370 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7945df5b5-pcgp6"] Oct 29 22:42:21.663481 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.663422 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78f4dff4c9-87jff"] Oct 29 22:42:21.664997 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.664973 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-868dt"] Oct 29 22:42:21.665101 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.665021 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gj6vs"] Oct 29 22:42:21.680616 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.680590 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-lm55w"] Oct 29 22:42:21.687379 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.687352 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b"] Oct 29 22:42:21.688487 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.688455 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qbv9d"] Oct 29 22:42:21.689490 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.689466 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-kw5c9"] Oct 29 22:42:21.714008 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.713986 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 29 22:42:21.714008 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.714000 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 29 22:42:21.714008 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.713987 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:42:21.714303 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.714000 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 29 22:42:21.714303 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.713987 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 29 22:42:21.714303 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.714078 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 29 22:42:21.714303 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.714115 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:42:21.714303 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.714127 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 29 22:42:21.714303 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.714173 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 29 22:42:21.714303 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.714199 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 29 22:42:21.714303 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.714228 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:42:21.714303 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.714270 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 29 22:42:21.714303 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.714324 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 22:42:21.714911 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.714388 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 29 22:42:21.714911 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.714414 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 22:42:21.714911 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.714482 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 29 22:42:21.715463 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.715445 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 22:42:21.715572 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.715514 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 22:42:21.715572 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.715523 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:42:21.715572 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.715517 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 29 22:42:21.715572 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.715528 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 29 22:42:21.715572 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.715555 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 29 22:42:21.715572 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.715557 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 29 22:42:21.715572 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.715568 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 22:42:21.715898 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.715581 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 22:42:21.726092 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.726075 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 29 22:42:21.726759 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.726738 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/884370ff-c54f-4d6b-af78-5c0813bb48eb-installation-pull-secrets\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.726829 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.726774 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5l7q\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-kube-api-access-t5l7q\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.726829 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.726803 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e91a303-9b80-47f5-a502-fbb6365eaa54-config\") pod \"console-operator-7cd56b8dbc-lm55w\" (UID: \"9e91a303-9b80-47f5-a502-fbb6365eaa54\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:21.726940 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.726830 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 22:42:21.726940 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.726868 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-bound-sa-token\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.726940 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.726921 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m76zp\" (UniqueName: \"kubernetes.io/projected/8f6c01b9-e044-413c-b533-09033d04f29e-kube-api-access-m76zp\") pod \"volume-data-source-validator-58c9f5f46d-868dt\" (UID: \"8f6c01b9-e044-413c-b533-09033d04f29e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-868dt" Oct 29 22:42:21.727040 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.726953 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:42:21.727040 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.726974 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-tmp\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:21.727040 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727008 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82r6j\" (UniqueName: \"kubernetes.io/projected/266bff54-6027-4822-91ce-be5cbd316ddc-kube-api-access-82r6j\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:21.727124 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727081 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6f91740-e20b-439e-be7d-4b444f4835fa-cert\") pod \"ingress-canary-qbv9d\" (UID: \"d6f91740-e20b-439e-be7d-4b444f4835fa\") " pod="openshift-ingress-canary/ingress-canary-qbv9d" Oct 29 22:42:21.727124 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727107 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-snapshots\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:21.727196 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727132 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/093ceb90-294f-4133-83a6-32548405d530-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7klbh\" (UID: \"093ceb90-294f-4133-83a6-32548405d530\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" Oct 29 22:42:21.727196 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727164 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-stats-auth\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:21.727261 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727219 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9l7w\" (UniqueName: \"kubernetes.io/projected/85ab246f-f679-40c5-a127-c0fd5a4a621e-kube-api-access-g9l7w\") pod \"network-check-source-6f78d59f99-kw5c9\" (UID: \"85ab246f-f679-40c5-a127-c0fd5a4a621e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kw5c9" Oct 29 22:42:21.727261 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727238 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgc6n\" (UniqueName: \"kubernetes.io/projected/a1a232f4-f67c-404f-a1f7-af0c3cc30180-kube-api-access-kgc6n\") pod \"kas-connection-checker-685895c7c9-fh7rx\" (UID: \"a1a232f4-f67c-404f-a1f7-af0c3cc30180\") " pod="kube-system/kas-connection-checker-685895c7c9-fh7rx" Oct 29 22:42:21.727261 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727259 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.727347 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727295 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cffk\" (UniqueName: \"kubernetes.io/projected/093ceb90-294f-4133-83a6-32548405d530-kube-api-access-5cffk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7klbh\" (UID: \"093ceb90-294f-4133-83a6-32548405d530\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" Oct 29 22:42:21.727347 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727334 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/877326a8-91a3-499f-8b37-4f80dfb00352-ca-trust-extracted\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.727404 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727355 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:21.727404 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727385 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e91a303-9b80-47f5-a502-fbb6365eaa54-serving-cert\") pod \"console-operator-7cd56b8dbc-lm55w\" (UID: \"9e91a303-9b80-47f5-a502-fbb6365eaa54\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:21.727462 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727407 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-default-certificate\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:21.727462 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727458 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz8wx\" (UniqueName: \"kubernetes.io/projected/68daf26a-81a0-4bd5-a188-9b39a1f460a7-kube-api-access-rz8wx\") pod \"kas-connection-checker-685895c7c9-k8xnq\" (UID: \"68daf26a-81a0-4bd5-a188-9b39a1f460a7\") " pod="kube-system/kas-connection-checker-685895c7c9-k8xnq" Oct 29 22:42:21.727518 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727479 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-serving-cert\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:21.727555 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727527 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/877326a8-91a3-499f-8b37-4f80dfb00352-image-registry-private-configuration\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.727555 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727544 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e91a303-9b80-47f5-a502-fbb6365eaa54-trusted-ca\") pod \"console-operator-7cd56b8dbc-lm55w\" (UID: \"9e91a303-9b80-47f5-a502-fbb6365eaa54\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:21.727628 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727565 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqxqz\" (UniqueName: \"kubernetes.io/projected/a109715c-c29a-48d1-a573-3e7800f48259-kube-api-access-zqxqz\") pod \"service-ca-operator-544d749c8c-j74gf\" (UID: \"a109715c-c29a-48d1-a573-3e7800f48259\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" Oct 29 22:42:21.727628 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727586 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:21.727709 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727640 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg6m4\" (UniqueName: \"kubernetes.io/projected/96726184-9be2-4685-93d1-415bbb8b551c-kube-api-access-cg6m4\") pod \"cluster-samples-operator-6ddb6c8b94-8ncgh\" (UID: \"96726184-9be2-4685-93d1-415bbb8b551c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:21.727709 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727684 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kpn4\" (UniqueName: \"kubernetes.io/projected/ad00084a-5a9f-489b-9122-2316f617f3f1-kube-api-access-2kpn4\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:21.727786 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727720 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99577b75-29cd-430b-b5ac-2af3828586d0-service-ca-bundle\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:21.727823 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727788 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-bound-sa-token\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.727823 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727815 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a109715c-c29a-48d1-a573-3e7800f48259-config\") pod \"service-ca-operator-544d749c8c-j74gf\" (UID: \"a109715c-c29a-48d1-a573-3e7800f48259\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" Oct 29 22:42:21.727908 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727831 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6nxm\" (UniqueName: \"kubernetes.io/projected/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-kube-api-access-f6nxm\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:21.727908 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727846 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:21.727908 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727862 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/877326a8-91a3-499f-8b37-4f80dfb00352-installation-pull-secrets\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.728031 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727935 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/093ceb90-294f-4133-83a6-32548405d530-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7klbh\" (UID: \"093ceb90-294f-4133-83a6-32548405d530\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" Oct 29 22:42:21.728031 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.727973 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/575ee3c4-7c77-4b31-9961-eebc60254d19-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rwfmh\" (UID: \"575ee3c4-7c77-4b31-9961-eebc60254d19\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rwfmh" Oct 29 22:42:21.728104 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728024 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/877326a8-91a3-499f-8b37-4f80dfb00352-trusted-ca\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.728104 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728093 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/877326a8-91a3-499f-8b37-4f80dfb00352-registry-certificates\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.728173 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728138 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcjhj\" (UniqueName: \"kubernetes.io/projected/6a4e4c73-0516-464c-9b6e-1c4adf93c03b-kube-api-access-dcjhj\") pod \"kas-connection-checker-685895c7c9-7hz4v\" (UID: \"6a4e4c73-0516-464c-9b6e-1c4adf93c03b\") " pod="kube-system/kas-connection-checker-685895c7c9-7hz4v" Oct 29 22:42:21.728173 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728166 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ad00084a-5a9f-489b-9122-2316f617f3f1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:21.728280 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728228 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-metrics-certs\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:21.728280 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728276 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcfvz\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-kube-api-access-hcfvz\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.728383 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728314 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/266bff54-6027-4822-91ce-be5cbd316ddc-config-volume\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:21.728383 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728341 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/266bff54-6027-4822-91ce-be5cbd316ddc-tmp-dir\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:21.728480 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728379 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-certificates\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.728480 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728413 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-service-ca-bundle\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:21.728480 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728448 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx2rx\" (UniqueName: \"kubernetes.io/projected/d6f91740-e20b-439e-be7d-4b444f4835fa-kube-api-access-rx2rx\") pod \"ingress-canary-qbv9d\" (UID: \"d6f91740-e20b-439e-be7d-4b444f4835fa\") " pod="openshift-ingress-canary/ingress-canary-qbv9d" Oct 29 22:42:21.728480 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728473 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.728593 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728489 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/884370ff-c54f-4d6b-af78-5c0813bb48eb-trusted-ca\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.728593 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728505 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a109715c-c29a-48d1-a573-3e7800f48259-serving-cert\") pod \"service-ca-operator-544d749c8c-j74gf\" (UID: \"a109715c-c29a-48d1-a573-3e7800f48259\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" Oct 29 22:42:21.728593 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728522 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvm7r\" (UniqueName: \"kubernetes.io/projected/9e91a303-9b80-47f5-a502-fbb6365eaa54-kube-api-access-zvm7r\") pod \"console-operator-7cd56b8dbc-lm55w\" (UID: \"9e91a303-9b80-47f5-a502-fbb6365eaa54\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:21.728593 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728585 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/575ee3c4-7c77-4b31-9961-eebc60254d19-nginx-conf\") pod \"networking-console-plugin-86c478bb88-rwfmh\" (UID: \"575ee3c4-7c77-4b31-9961-eebc60254d19\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rwfmh" Oct 29 22:42:21.728733 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728603 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-8ncgh\" (UID: \"96726184-9be2-4685-93d1-415bbb8b551c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:21.728733 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728620 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h744b\" (UniqueName: \"kubernetes.io/projected/99577b75-29cd-430b-b5ac-2af3828586d0-kube-api-access-h744b\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:21.728733 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728693 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/884370ff-c54f-4d6b-af78-5c0813bb48eb-image-registry-private-configuration\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.728733 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728725 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8ncgh\" (UID: \"96726184-9be2-4685-93d1-415bbb8b551c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:21.728937 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.728745 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/884370ff-c54f-4d6b-af78-5c0813bb48eb-ca-trust-extracted\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.734713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.734692 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 29 22:42:21.734824 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.734762 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 29 22:42:21.734896 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.734825 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 29 22:42:21.734896 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.734849 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 29 22:42:21.735028 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.735012 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 29 22:42:21.735083 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.735017 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 29 22:42:21.735083 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.735026 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 29 22:42:21.735083 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.735037 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 29 22:42:21.735083 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.735051 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 29 22:42:21.735240 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.735138 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 29 22:42:21.735240 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.735177 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 29 22:42:21.735365 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.735351 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 29 22:42:21.735418 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.735390 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 22:42:21.735418 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.735409 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nqrw2\"" Oct 29 22:42:21.735496 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.735410 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-m9r9w\"" Oct 29 22:42:21.735496 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.735427 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-p26v7\"" Oct 29 22:42:21.735496 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.735433 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6l2xz\"" Oct 29 22:42:21.735496 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.735439 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6m48j\"" Oct 29 22:42:21.735496 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.735428 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 22:42:21.735496 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.735456 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-gpfcv\"" Oct 29 22:42:21.735496 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.735488 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6752d\"" Oct 29 22:42:21.735688 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.735545 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vscck\"" Oct 29 22:42:21.742031 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.742008 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-65zqz\"" Oct 29 22:42:21.742031 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.742017 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-h9lx2\"" Oct 29 22:42:21.742031 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.742017 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d6hqb\"" Oct 29 22:42:21.742216 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.742047 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vllbq\"" Oct 29 22:42:21.751722 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.751654 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 29 22:42:21.751722 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.751713 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 29 22:42:21.752142 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.751761 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 29 22:42:21.752537 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.752513 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 29 22:42:21.752800 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.752783 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:21.753324 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.753304 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 29 22:42:21.819788 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.819752 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 22:42:21.819788 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.819761 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kfq4\"" Oct 29 22:42:21.829075 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829050 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-snapshots\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:21.829131 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829082 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/093ceb90-294f-4133-83a6-32548405d530-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7klbh\" (UID: \"093ceb90-294f-4133-83a6-32548405d530\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" Oct 29 22:42:21.829131 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829098 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-stats-auth\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:21.829131 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829123 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9l7w\" (UniqueName: \"kubernetes.io/projected/85ab246f-f679-40c5-a127-c0fd5a4a621e-kube-api-access-g9l7w\") pod \"network-check-source-6f78d59f99-kw5c9\" (UID: \"85ab246f-f679-40c5-a127-c0fd5a4a621e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kw5c9" Oct 29 22:42:21.829283 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829150 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgc6n\" (UniqueName: \"kubernetes.io/projected/a1a232f4-f67c-404f-a1f7-af0c3cc30180-kube-api-access-kgc6n\") pod \"kas-connection-checker-685895c7c9-fh7rx\" (UID: \"a1a232f4-f67c-404f-a1f7-af0c3cc30180\") " pod="kube-system/kas-connection-checker-685895c7c9-fh7rx" Oct 29 22:42:21.829283 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829177 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.829283 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829222 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cffk\" (UniqueName: \"kubernetes.io/projected/093ceb90-294f-4133-83a6-32548405d530-kube-api-access-5cffk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7klbh\" (UID: \"093ceb90-294f-4133-83a6-32548405d530\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" Oct 29 22:42:21.829283 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829248 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/877326a8-91a3-499f-8b37-4f80dfb00352-ca-trust-extracted\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.829283 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829272 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:21.829496 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829294 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e91a303-9b80-47f5-a502-fbb6365eaa54-serving-cert\") pod \"console-operator-7cd56b8dbc-lm55w\" (UID: \"9e91a303-9b80-47f5-a502-fbb6365eaa54\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:21.829496 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829317 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-default-certificate\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:21.829496 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829342 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz8wx\" (UniqueName: \"kubernetes.io/projected/68daf26a-81a0-4bd5-a188-9b39a1f460a7-kube-api-access-rz8wx\") pod \"kas-connection-checker-685895c7c9-k8xnq\" (UID: \"68daf26a-81a0-4bd5-a188-9b39a1f460a7\") " pod="kube-system/kas-connection-checker-685895c7c9-k8xnq" Oct 29 22:42:21.829496 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829361 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-serving-cert\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:21.829496 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829385 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/877326a8-91a3-499f-8b37-4f80dfb00352-image-registry-private-configuration\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.829496 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829405 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e91a303-9b80-47f5-a502-fbb6365eaa54-trusted-ca\") pod \"console-operator-7cd56b8dbc-lm55w\" (UID: \"9e91a303-9b80-47f5-a502-fbb6365eaa54\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:21.829496 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829470 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqxqz\" (UniqueName: \"kubernetes.io/projected/a109715c-c29a-48d1-a573-3e7800f48259-kube-api-access-zqxqz\") pod \"service-ca-operator-544d749c8c-j74gf\" (UID: \"a109715c-c29a-48d1-a573-3e7800f48259\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" Oct 29 22:42:21.829496 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829500 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:21.829868 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829529 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg6m4\" (UniqueName: \"kubernetes.io/projected/96726184-9be2-4685-93d1-415bbb8b551c-kube-api-access-cg6m4\") pod \"cluster-samples-operator-6ddb6c8b94-8ncgh\" (UID: \"96726184-9be2-4685-93d1-415bbb8b551c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:21.829868 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829558 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kpn4\" (UniqueName: \"kubernetes.io/projected/ad00084a-5a9f-489b-9122-2316f617f3f1-kube-api-access-2kpn4\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:21.829868 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829583 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99577b75-29cd-430b-b5ac-2af3828586d0-service-ca-bundle\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:21.829868 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829608 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-bound-sa-token\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.829868 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829687 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/093ceb90-294f-4133-83a6-32548405d530-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7klbh\" (UID: \"093ceb90-294f-4133-83a6-32548405d530\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" Oct 29 22:42:21.829868 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829821 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-snapshots\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:21.830242 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829695 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a109715c-c29a-48d1-a573-3e7800f48259-config\") pod \"service-ca-operator-544d749c8c-j74gf\" (UID: \"a109715c-c29a-48d1-a573-3e7800f48259\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" Oct 29 22:42:21.830242 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829941 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6nxm\" (UniqueName: \"kubernetes.io/projected/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-kube-api-access-f6nxm\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:21.830242 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.829973 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:21.830242 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830010 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/877326a8-91a3-499f-8b37-4f80dfb00352-installation-pull-secrets\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.830242 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830039 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/093ceb90-294f-4133-83a6-32548405d530-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7klbh\" (UID: \"093ceb90-294f-4133-83a6-32548405d530\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" Oct 29 22:42:21.830242 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/575ee3c4-7c77-4b31-9961-eebc60254d19-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rwfmh\" (UID: \"575ee3c4-7c77-4b31-9961-eebc60254d19\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rwfmh" Oct 29 22:42:21.830242 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830091 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/877326a8-91a3-499f-8b37-4f80dfb00352-trusted-ca\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.830242 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830120 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/877326a8-91a3-499f-8b37-4f80dfb00352-registry-certificates\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.830242 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830148 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcjhj\" (UniqueName: \"kubernetes.io/projected/6a4e4c73-0516-464c-9b6e-1c4adf93c03b-kube-api-access-dcjhj\") pod \"kas-connection-checker-685895c7c9-7hz4v\" (UID: \"6a4e4c73-0516-464c-9b6e-1c4adf93c03b\") " pod="kube-system/kas-connection-checker-685895c7c9-7hz4v" Oct 29 22:42:21.830242 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830174 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ad00084a-5a9f-489b-9122-2316f617f3f1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:21.830242 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830214 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-metrics-certs\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:21.830833 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830252 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcfvz\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-kube-api-access-hcfvz\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.830833 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830280 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/266bff54-6027-4822-91ce-be5cbd316ddc-config-volume\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:21.830833 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830304 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/266bff54-6027-4822-91ce-be5cbd316ddc-tmp-dir\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:21.830833 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830342 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-certificates\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.830833 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830368 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-service-ca-bundle\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:21.830833 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830398 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx2rx\" (UniqueName: \"kubernetes.io/projected/d6f91740-e20b-439e-be7d-4b444f4835fa-kube-api-access-rx2rx\") pod \"ingress-canary-qbv9d\" (UID: \"d6f91740-e20b-439e-be7d-4b444f4835fa\") " pod="openshift-ingress-canary/ingress-canary-qbv9d" Oct 29 22:42:21.830833 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830423 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.830833 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830445 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/884370ff-c54f-4d6b-af78-5c0813bb48eb-trusted-ca\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.830833 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830547 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a109715c-c29a-48d1-a573-3e7800f48259-serving-cert\") pod \"service-ca-operator-544d749c8c-j74gf\" (UID: \"a109715c-c29a-48d1-a573-3e7800f48259\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" Oct 29 22:42:21.830833 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830574 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvm7r\" (UniqueName: \"kubernetes.io/projected/9e91a303-9b80-47f5-a502-fbb6365eaa54-kube-api-access-zvm7r\") pod \"console-operator-7cd56b8dbc-lm55w\" (UID: \"9e91a303-9b80-47f5-a502-fbb6365eaa54\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:21.830833 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:21.830576 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:42:21.830833 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:21.830594 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf56cb474-gx24x: secret "image-registry-tls" not found Oct 29 22:42:21.830833 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:21.830659 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls podName:877326a8-91a3-499f-8b37-4f80dfb00352 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:22.330640615 +0000 UTC m=+29.069774703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls") pod "image-registry-5cf56cb474-gx24x" (UID: "877326a8-91a3-499f-8b37-4f80dfb00352") : secret "image-registry-tls" not found Oct 29 22:42:21.830833 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830684 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/575ee3c4-7c77-4b31-9961-eebc60254d19-nginx-conf\") pod \"networking-console-plugin-86c478bb88-rwfmh\" (UID: \"575ee3c4-7c77-4b31-9961-eebc60254d19\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rwfmh" Oct 29 22:42:21.830833 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830722 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-8ncgh\" (UID: \"96726184-9be2-4685-93d1-415bbb8b551c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:21.830833 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830753 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h744b\" (UniqueName: \"kubernetes.io/projected/99577b75-29cd-430b-b5ac-2af3828586d0-kube-api-access-h744b\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:21.832158 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830784 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/884370ff-c54f-4d6b-af78-5c0813bb48eb-image-registry-private-configuration\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.832158 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830821 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8ncgh\" (UID: \"96726184-9be2-4685-93d1-415bbb8b551c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:21.832158 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830849 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/884370ff-c54f-4d6b-af78-5c0813bb48eb-ca-trust-extracted\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.832158 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830867 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:21.832158 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.830918 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/884370ff-c54f-4d6b-af78-5c0813bb48eb-installation-pull-secrets\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.832158 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:21.831092 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 22:42:21.832158 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:21.831136 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls podName:266bff54-6027-4822-91ce-be5cbd316ddc nodeName:}" failed. No retries permitted until 2026-10-29 22:42:22.331122903 +0000 UTC m=+29.070256990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls") pod "dns-default-gj6vs" (UID: "266bff54-6027-4822-91ce-be5cbd316ddc") : secret "dns-default-metrics-tls" not found Oct 29 22:42:21.832158 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.831518 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5l7q\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-kube-api-access-t5l7q\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.832158 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.831558 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e91a303-9b80-47f5-a502-fbb6365eaa54-config\") pod \"console-operator-7cd56b8dbc-lm55w\" (UID: \"9e91a303-9b80-47f5-a502-fbb6365eaa54\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:21.832158 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.831607 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-bound-sa-token\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.832158 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.831636 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m76zp\" (UniqueName: \"kubernetes.io/projected/8f6c01b9-e044-413c-b533-09033d04f29e-kube-api-access-m76zp\") pod \"volume-data-source-validator-58c9f5f46d-868dt\" (UID: \"8f6c01b9-e044-413c-b533-09033d04f29e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-868dt" Oct 29 22:42:21.832158 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.831663 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-tmp\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:21.832158 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.831691 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82r6j\" (UniqueName: \"kubernetes.io/projected/266bff54-6027-4822-91ce-be5cbd316ddc-kube-api-access-82r6j\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:21.832158 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.831733 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6f91740-e20b-439e-be7d-4b444f4835fa-cert\") pod \"ingress-canary-qbv9d\" (UID: \"d6f91740-e20b-439e-be7d-4b444f4835fa\") " pod="openshift-ingress-canary/ingress-canary-qbv9d" Oct 29 22:42:21.832158 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.831790 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/575ee3c4-7c77-4b31-9961-eebc60254d19-nginx-conf\") pod \"networking-console-plugin-86c478bb88-rwfmh\" (UID: \"575ee3c4-7c77-4b31-9961-eebc60254d19\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rwfmh" Oct 29 22:42:21.832158 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.831930 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a109715c-c29a-48d1-a573-3e7800f48259-config\") pod \"service-ca-operator-544d749c8c-j74gf\" (UID: \"a109715c-c29a-48d1-a573-3e7800f48259\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" Oct 29 22:42:21.833463 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.833058 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/877326a8-91a3-499f-8b37-4f80dfb00352-ca-trust-extracted\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.833463 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:21.833151 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:42:21.833463 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:21.833225 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls podName:ad00084a-5a9f-489b-9122-2316f617f3f1 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:22.333209754 +0000 UTC m=+29.072343834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dzx6b" (UID: "ad00084a-5a9f-489b-9122-2316f617f3f1") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:42:21.833463 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:21.833402 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 22:42:21.833463 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:21.833449 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-tls podName:96726184-9be2-4685-93d1-415bbb8b551c nodeName:}" failed. No retries permitted until 2026-10-29 22:42:22.333434302 +0000 UTC m=+29.072568393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8ncgh" (UID: "96726184-9be2-4685-93d1-415bbb8b551c") : secret "samples-operator-tls" not found Oct 29 22:42:21.833733 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:21.833552 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99577b75-29cd-430b-b5ac-2af3828586d0-service-ca-bundle podName:99577b75-29cd-430b-b5ac-2af3828586d0 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:22.333533324 +0000 UTC m=+29.072667418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99577b75-29cd-430b-b5ac-2af3828586d0-service-ca-bundle") pod "router-default-7945df5b5-pcgp6" (UID: "99577b75-29cd-430b-b5ac-2af3828586d0") : configmap references non-existent config key: service-ca.crt Oct 29 22:42:21.833794 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.833773 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/884370ff-c54f-4d6b-af78-5c0813bb48eb-ca-trust-extracted\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.834146 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.834122 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-serving-cert\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:21.834259 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.834156 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e91a303-9b80-47f5-a502-fbb6365eaa54-config\") pod \"console-operator-7cd56b8dbc-lm55w\" (UID: \"9e91a303-9b80-47f5-a502-fbb6365eaa54\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:21.834259 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.834219 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-tmp\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:21.834850 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:21.834453 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:42:21.834850 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:21.834505 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/575ee3c4-7c77-4b31-9961-eebc60254d19-networking-console-plugin-cert podName:575ee3c4-7c77-4b31-9961-eebc60254d19 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:22.334490483 +0000 UTC m=+29.073624562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/575ee3c4-7c77-4b31-9961-eebc60254d19-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-rwfmh" (UID: "575ee3c4-7c77-4b31-9961-eebc60254d19") : secret "networking-console-plugin-cert" not found Oct 29 22:42:21.835001 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.834877 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/266bff54-6027-4822-91ce-be5cbd316ddc-tmp-dir\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:21.835059 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.834997 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e91a303-9b80-47f5-a502-fbb6365eaa54-trusted-ca\") pod \"console-operator-7cd56b8dbc-lm55w\" (UID: \"9e91a303-9b80-47f5-a502-fbb6365eaa54\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:21.836323 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.835354 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e91a303-9b80-47f5-a502-fbb6365eaa54-serving-cert\") pod \"console-operator-7cd56b8dbc-lm55w\" (UID: \"9e91a303-9b80-47f5-a502-fbb6365eaa54\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:21.836323 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.835368 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-certificates\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.836323 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.835478 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/877326a8-91a3-499f-8b37-4f80dfb00352-registry-certificates\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.836323 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.835635 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-8ncgh\" (UID: \"96726184-9be2-4685-93d1-415bbb8b551c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:21.836323 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.835728 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/877326a8-91a3-499f-8b37-4f80dfb00352-installation-pull-secrets\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.836323 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.835796 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/877326a8-91a3-499f-8b37-4f80dfb00352-trusted-ca\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.836323 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.835837 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-service-ca-bundle\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:21.836323 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:21.835854 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:42:21.836323 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:21.835870 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f4dff4c9-87jff: secret "image-registry-tls" not found Oct 29 22:42:21.836323 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:21.835961 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls podName:884370ff-c54f-4d6b-af78-5c0813bb48eb nodeName:}" failed. No retries permitted until 2026-10-29 22:42:22.335946665 +0000 UTC m=+29.075080761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls") pod "image-registry-78f4dff4c9-87jff" (UID: "884370ff-c54f-4d6b-af78-5c0813bb48eb") : secret "image-registry-tls" not found Oct 29 22:42:21.836323 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:21.836258 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 22:42:21.836323 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:21.836326 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-metrics-certs podName:99577b75-29cd-430b-b5ac-2af3828586d0 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:22.336310206 +0000 UTC m=+29.075444302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-metrics-certs") pod "router-default-7945df5b5-pcgp6" (UID: "99577b75-29cd-430b-b5ac-2af3828586d0") : secret "router-metrics-certs-default" not found Oct 29 22:42:21.836940 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.836639 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ad00084a-5a9f-489b-9122-2316f617f3f1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:21.836940 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.836722 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/884370ff-c54f-4d6b-af78-5c0813bb48eb-trusted-ca\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.836940 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.836743 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-stats-auth\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:21.836940 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.836762 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6f91740-e20b-439e-be7d-4b444f4835fa-cert\") pod \"ingress-canary-qbv9d\" (UID: \"d6f91740-e20b-439e-be7d-4b444f4835fa\") " pod="openshift-ingress-canary/ingress-canary-qbv9d" Oct 29 22:42:21.837233 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.837207 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/266bff54-6027-4822-91ce-be5cbd316ddc-config-volume\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:21.837326 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.837304 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/884370ff-c54f-4d6b-af78-5c0813bb48eb-image-registry-private-configuration\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.838853 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.838811 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/093ceb90-294f-4133-83a6-32548405d530-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7klbh\" (UID: \"093ceb90-294f-4133-83a6-32548405d530\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" Oct 29 22:42:21.839015 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.838963 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/877326a8-91a3-499f-8b37-4f80dfb00352-image-registry-private-configuration\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.839263 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.839243 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a109715c-c29a-48d1-a573-3e7800f48259-serving-cert\") pod \"service-ca-operator-544d749c8c-j74gf\" (UID: \"a109715c-c29a-48d1-a573-3e7800f48259\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" Oct 29 22:42:21.839490 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.839473 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/884370ff-c54f-4d6b-af78-5c0813bb48eb-installation-pull-secrets\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.840206 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.840169 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-default-certificate\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:21.855150 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.855117 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9l7w\" (UniqueName: \"kubernetes.io/projected/85ab246f-f679-40c5-a127-c0fd5a4a621e-kube-api-access-g9l7w\") pod \"network-check-source-6f78d59f99-kw5c9\" (UID: \"85ab246f-f679-40c5-a127-c0fd5a4a621e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kw5c9" Oct 29 22:42:21.857149 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.857118 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cffk\" (UniqueName: \"kubernetes.io/projected/093ceb90-294f-4133-83a6-32548405d530-kube-api-access-5cffk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7klbh\" (UID: \"093ceb90-294f-4133-83a6-32548405d530\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" Oct 29 22:42:21.857458 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.857438 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz8wx\" (UniqueName: \"kubernetes.io/projected/68daf26a-81a0-4bd5-a188-9b39a1f460a7-kube-api-access-rz8wx\") pod \"kas-connection-checker-685895c7c9-k8xnq\" (UID: \"68daf26a-81a0-4bd5-a188-9b39a1f460a7\") " pod="kube-system/kas-connection-checker-685895c7c9-k8xnq" Oct 29 22:42:21.857645 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.857623 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kpn4\" (UniqueName: \"kubernetes.io/projected/ad00084a-5a9f-489b-9122-2316f617f3f1-kube-api-access-2kpn4\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:21.858863 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.858820 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-bound-sa-token\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.860325 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.860297 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqxqz\" (UniqueName: \"kubernetes.io/projected/a109715c-c29a-48d1-a573-3e7800f48259-kube-api-access-zqxqz\") pod \"service-ca-operator-544d749c8c-j74gf\" (UID: \"a109715c-c29a-48d1-a573-3e7800f48259\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" Oct 29 22:42:21.860797 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.860768 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvm7r\" (UniqueName: \"kubernetes.io/projected/9e91a303-9b80-47f5-a502-fbb6365eaa54-kube-api-access-zvm7r\") pod \"console-operator-7cd56b8dbc-lm55w\" (UID: \"9e91a303-9b80-47f5-a502-fbb6365eaa54\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:21.861309 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.861285 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6nxm\" (UniqueName: \"kubernetes.io/projected/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-kube-api-access-f6nxm\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:21.861405 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.861382 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgc6n\" (UniqueName: \"kubernetes.io/projected/a1a232f4-f67c-404f-a1f7-af0c3cc30180-kube-api-access-kgc6n\") pod \"kas-connection-checker-685895c7c9-fh7rx\" (UID: \"a1a232f4-f67c-404f-a1f7-af0c3cc30180\") " pod="kube-system/kas-connection-checker-685895c7c9-fh7rx" Oct 29 22:42:21.861614 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.861596 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg6m4\" (UniqueName: \"kubernetes.io/projected/96726184-9be2-4685-93d1-415bbb8b551c-kube-api-access-cg6m4\") pod \"cluster-samples-operator-6ddb6c8b94-8ncgh\" (UID: \"96726184-9be2-4685-93d1-415bbb8b551c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:21.863452 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.863430 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m76zp\" (UniqueName: \"kubernetes.io/projected/8f6c01b9-e044-413c-b533-09033d04f29e-kube-api-access-m76zp\") pod \"volume-data-source-validator-58c9f5f46d-868dt\" (UID: \"8f6c01b9-e044-413c-b533-09033d04f29e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-868dt" Oct 29 22:42:21.863891 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.863867 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5l7q\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-kube-api-access-t5l7q\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:21.864930 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.864909 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcjhj\" (UniqueName: \"kubernetes.io/projected/6a4e4c73-0516-464c-9b6e-1c4adf93c03b-kube-api-access-dcjhj\") pod \"kas-connection-checker-685895c7c9-7hz4v\" (UID: \"6a4e4c73-0516-464c-9b6e-1c4adf93c03b\") " pod="kube-system/kas-connection-checker-685895c7c9-7hz4v" Oct 29 22:42:21.865027 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.865011 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82r6j\" (UniqueName: \"kubernetes.io/projected/266bff54-6027-4822-91ce-be5cbd316ddc-kube-api-access-82r6j\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:21.865708 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.865692 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-bound-sa-token\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.866629 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.866615 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h744b\" (UniqueName: \"kubernetes.io/projected/99577b75-29cd-430b-b5ac-2af3828586d0-kube-api-access-h744b\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:21.875973 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.875947 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx2rx\" (UniqueName: \"kubernetes.io/projected/d6f91740-e20b-439e-be7d-4b444f4835fa-kube-api-access-rx2rx\") pod \"ingress-canary-qbv9d\" (UID: \"d6f91740-e20b-439e-be7d-4b444f4835fa\") " pod="openshift-ingress-canary/ingress-canary-qbv9d" Oct 29 22:42:21.876204 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.876167 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcfvz\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-kube-api-access-hcfvz\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:21.902500 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.902460 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7hz4v" Oct 29 22:42:21.913523 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.913498 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fh7rx" Oct 29 22:42:21.921480 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.921453 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-k8xnq" Oct 29 22:42:21.928726 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.928700 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" Oct 29 22:42:21.935158 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.935076 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-868dt" Oct 29 22:42:21.991387 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:21.991113 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:22.003604 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.003368 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" Oct 29 22:42:22.011810 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.011465 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:22.019001 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.018955 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kw5c9" Oct 29 22:42:22.029564 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.029386 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qbv9d" Oct 29 22:42:22.230498 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.230328 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fh7rx"] Oct 29 22:42:22.232240 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:42:22.232211 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a232f4_f67c_404f_a1f7_af0c3cc30180.slice/crio-9870c7f5e182878c16f0d6c428a1c62d407777effb29d920c0ffe042b0ed2873 WatchSource:0}: Error finding container 9870c7f5e182878c16f0d6c428a1c62d407777effb29d920c0ffe042b0ed2873: Status 404 returned error can't find the container with id 9870c7f5e182878c16f0d6c428a1c62d407777effb29d920c0ffe042b0ed2873 Oct 29 22:42:22.234262 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.234243 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7hz4v"] Oct 29 22:42:22.235648 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:42:22.235620 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4e4c73_0516_464c_9b6e_1c4adf93c03b.slice/crio-e28f2f94763c91944b4b5e6fcdda5c2aa18ee62de01c0f5b3afbe0d2c67e887f WatchSource:0}: Error finding container e28f2f94763c91944b4b5e6fcdda5c2aa18ee62de01c0f5b3afbe0d2c67e887f: Status 404 returned error can't find the container with id e28f2f94763c91944b4b5e6fcdda5c2aa18ee62de01c0f5b3afbe0d2c67e887f Oct 29 22:42:22.238989 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.237390 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k8xnq"] Oct 29 22:42:22.239865 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:42:22.239815 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68daf26a_81a0_4bd5_a188_9b39a1f460a7.slice/crio-25d1297cee065faf2844f297078cbe34338e029f0120e4f6fbadc072a2e4f1e9 WatchSource:0}: Error finding container 25d1297cee065faf2844f297078cbe34338e029f0120e4f6fbadc072a2e4f1e9: Status 404 returned error can't find the container with id 25d1297cee065faf2844f297078cbe34338e029f0120e4f6fbadc072a2e4f1e9 Oct 29 22:42:22.244974 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.244956 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh"] Oct 29 22:42:22.245967 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:42:22.245946 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093ceb90_294f_4133_83a6_32548405d530.slice/crio-ac9dd7d8666ac55fac7d0ebe766e087035079f15b2f3dcffcfb734b1f0597a10 WatchSource:0}: Error finding container ac9dd7d8666ac55fac7d0ebe766e087035079f15b2f3dcffcfb734b1f0597a10: Status 404 returned error can't find the container with id ac9dd7d8666ac55fac7d0ebe766e087035079f15b2f3dcffcfb734b1f0597a10 Oct 29 22:42:22.282125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.282060 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-868dt"] Oct 29 22:42:22.283832 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:42:22.283791 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6c01b9_e044_413c_b533_09033d04f29e.slice/crio-b0e0c2bbab3f446ae654fa789a3fdca7a20acb188bbae830f145505f196b75fc WatchSource:0}: Error finding container b0e0c2bbab3f446ae654fa789a3fdca7a20acb188bbae830f145505f196b75fc: Status 404 returned error can't find the container with id b0e0c2bbab3f446ae654fa789a3fdca7a20acb188bbae830f145505f196b75fc Oct 29 22:42:22.337859 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.337824 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99577b75-29cd-430b-b5ac-2af3828586d0-service-ca-bundle\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:22.337999 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.337884 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:22.337999 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.337924 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/575ee3c4-7c77-4b31-9961-eebc60254d19-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rwfmh\" (UID: \"575ee3c4-7c77-4b31-9961-eebc60254d19\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rwfmh" Oct 29 22:42:22.337999 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.337958 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-metrics-certs\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:22.337999 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:22.337995 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99577b75-29cd-430b-b5ac-2af3828586d0-service-ca-bundle podName:99577b75-29cd-430b-b5ac-2af3828586d0 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:23.337976848 +0000 UTC m=+30.077110936 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99577b75-29cd-430b-b5ac-2af3828586d0-service-ca-bundle") pod "router-default-7945df5b5-pcgp6" (UID: "99577b75-29cd-430b-b5ac-2af3828586d0") : configmap references non-existent config key: service-ca.crt Oct 29 22:42:22.338230 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.338042 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:22.338230 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:22.338063 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 22:42:22.338230 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.338093 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8ncgh\" (UID: \"96726184-9be2-4685-93d1-415bbb8b551c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:22.338230 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:22.338122 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-metrics-certs podName:99577b75-29cd-430b-b5ac-2af3828586d0 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:23.338107885 +0000 UTC m=+30.077241976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-metrics-certs") pod "router-default-7945df5b5-pcgp6" (UID: "99577b75-29cd-430b-b5ac-2af3828586d0") : secret "router-metrics-certs-default" not found Oct 29 22:42:22.338230 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:22.338167 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 22:42:22.338230 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.338220 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:22.338498 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.338257 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:22.338498 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:22.338269 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:42:22.338498 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:22.338278 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f4dff4c9-87jff: secret "image-registry-tls" not found Oct 29 22:42:22.338498 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:22.338302 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls podName:884370ff-c54f-4d6b-af78-5c0813bb48eb nodeName:}" failed. No retries permitted until 2026-10-29 22:42:23.338292896 +0000 UTC m=+30.077426997 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls") pod "image-registry-78f4dff4c9-87jff" (UID: "884370ff-c54f-4d6b-af78-5c0813bb48eb") : secret "image-registry-tls" not found Oct 29 22:42:22.338498 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:22.338337 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:42:22.338498 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:22.338357 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/575ee3c4-7c77-4b31-9961-eebc60254d19-networking-console-plugin-cert podName:575ee3c4-7c77-4b31-9961-eebc60254d19 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:23.338350907 +0000 UTC m=+30.077485021 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/575ee3c4-7c77-4b31-9961-eebc60254d19-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-rwfmh" (UID: "575ee3c4-7c77-4b31-9961-eebc60254d19") : secret "networking-console-plugin-cert" not found Oct 29 22:42:22.338498 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:22.338373 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:42:22.338498 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:22.338388 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 22:42:22.338498 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:22.338412 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls podName:ad00084a-5a9f-489b-9122-2316f617f3f1 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:23.338399962 +0000 UTC m=+30.077534146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dzx6b" (UID: "ad00084a-5a9f-489b-9122-2316f617f3f1") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:42:22.338498 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:22.338429 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls podName:266bff54-6027-4822-91ce-be5cbd316ddc nodeName:}" failed. No retries permitted until 2026-10-29 22:42:23.338419622 +0000 UTC m=+30.077553702 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls") pod "dns-default-gj6vs" (UID: "266bff54-6027-4822-91ce-be5cbd316ddc") : secret "dns-default-metrics-tls" not found Oct 29 22:42:22.338498 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:22.338432 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:42:22.338498 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:22.338440 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf56cb474-gx24x: secret "image-registry-tls" not found Oct 29 22:42:22.338498 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:22.338445 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-tls podName:96726184-9be2-4685-93d1-415bbb8b551c nodeName:}" failed. No retries permitted until 2026-10-29 22:42:23.338436105 +0000 UTC m=+30.077570184 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8ncgh" (UID: "96726184-9be2-4685-93d1-415bbb8b551c") : secret "samples-operator-tls" not found Oct 29 22:42:22.338498 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:22.338457 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls podName:877326a8-91a3-499f-8b37-4f80dfb00352 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:23.338451506 +0000 UTC m=+30.077585613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls") pod "image-registry-5cf56cb474-gx24x" (UID: "877326a8-91a3-499f-8b37-4f80dfb00352") : secret "image-registry-tls" not found Oct 29 22:42:22.376432 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.376333 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf"] Oct 29 22:42:22.379690 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.379620 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qbv9d"] Oct 29 22:42:22.380788 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.380764 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-lm55w"] Oct 29 22:42:22.386647 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:42:22.386622 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e91a303_9b80_47f5_a502_fbb6365eaa54.slice/crio-3ce7e52911f12103f3bf53cd734767cbff8b8fbdede8860e494cbcfc6f5813b7 WatchSource:0}: Error finding container 3ce7e52911f12103f3bf53cd734767cbff8b8fbdede8860e494cbcfc6f5813b7: Status 404 returned error can't find the container with id 3ce7e52911f12103f3bf53cd734767cbff8b8fbdede8860e494cbcfc6f5813b7 Oct 29 22:42:22.387681 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:42:22.387471 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f91740_e20b_439e_be7d_4b444f4835fa.slice/crio-ed05e90ad4316ad2bd053ae3fe901f90d0c3fe8f02ae86066c4e4e9f2729f8a6 WatchSource:0}: Error finding container ed05e90ad4316ad2bd053ae3fe901f90d0c3fe8f02ae86066c4e4e9f2729f8a6: Status 404 returned error can't find the container with id ed05e90ad4316ad2bd053ae3fe901f90d0c3fe8f02ae86066c4e4e9f2729f8a6 Oct 29 22:42:22.393072 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.392465 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-kw5c9"] Oct 29 22:42:22.396039 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:42:22.396016 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ab246f_f679_40c5_a127_c0fd5a4a621e.slice/crio-220631aa8bf08160ea2ae099ede645cd10b43b7f584b3f3f159920391e1cc22d WatchSource:0}: Error finding container 220631aa8bf08160ea2ae099ede645cd10b43b7f584b3f3f159920391e1cc22d: Status 404 returned error can't find the container with id 220631aa8bf08160ea2ae099ede645cd10b43b7f584b3f3f159920391e1cc22d Oct 29 22:42:22.406361 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.406338 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-9qhv9"] Oct 29 22:42:22.409514 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:42:22.409492 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5c9f0f_7f65_4cbc_a95a_a92e52c18aa8.slice/crio-b12aec2fedb8f3264afa0c6e4e36f3cc62676554e9adc0e336775257632dcf72 WatchSource:0}: Error finding container b12aec2fedb8f3264afa0c6e4e36f3cc62676554e9adc0e336775257632dcf72: Status 404 returned error can't find the container with id b12aec2fedb8f3264afa0c6e4e36f3cc62676554e9adc0e336775257632dcf72 Oct 29 22:42:22.751576 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.751540 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:22.751927 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.751832 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:22.770912 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.770884 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mcv8v\"" Oct 29 22:42:22.771044 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.771037 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 22:42:22.771230 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:22.771142 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ltz9s\"" Oct 29 22:42:23.049631 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.048980 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" event={"ID":"a109715c-c29a-48d1-a573-3e7800f48259","Type":"ContainerStarted","Data":"8b4ef5a62c85870b52d54ad3ffd257dee8b1784d7fd5cbbf259f4e8855eb7afb"} Oct 29 22:42:23.051485 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.051452 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7hz4v" event={"ID":"6a4e4c73-0516-464c-9b6e-1c4adf93c03b","Type":"ContainerStarted","Data":"ea64b060dda72ce0a5640c2dc9ca2d9282192c79378c405c1404ef6c0f55d910"} Oct 29 22:42:23.051620 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.051506 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7hz4v" event={"ID":"6a4e4c73-0516-464c-9b6e-1c4adf93c03b","Type":"ContainerStarted","Data":"e28f2f94763c91944b4b5e6fcdda5c2aa18ee62de01c0f5b3afbe0d2c67e887f"} Oct 29 22:42:23.053761 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.053718 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qbv9d" event={"ID":"d6f91740-e20b-439e-be7d-4b444f4835fa","Type":"ContainerStarted","Data":"ed05e90ad4316ad2bd053ae3fe901f90d0c3fe8f02ae86066c4e4e9f2729f8a6"} Oct 29 22:42:23.055676 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.055651 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" event={"ID":"9e91a303-9b80-47f5-a502-fbb6365eaa54","Type":"ContainerStarted","Data":"3ce7e52911f12103f3bf53cd734767cbff8b8fbdede8860e494cbcfc6f5813b7"} Oct 29 22:42:23.057559 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.057534 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kw5c9" event={"ID":"85ab246f-f679-40c5-a127-c0fd5a4a621e","Type":"ContainerStarted","Data":"220631aa8bf08160ea2ae099ede645cd10b43b7f584b3f3f159920391e1cc22d"} Oct 29 22:42:23.063002 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.062415 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-k8xnq" event={"ID":"68daf26a-81a0-4bd5-a188-9b39a1f460a7","Type":"ContainerStarted","Data":"6335d21ec0ec757b9fb278e424bfcc56d6c9b7df2b7b04bc90d482945d67d696"} Oct 29 22:42:23.063002 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.062458 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-k8xnq" event={"ID":"68daf26a-81a0-4bd5-a188-9b39a1f460a7","Type":"ContainerStarted","Data":"25d1297cee065faf2844f297078cbe34338e029f0120e4f6fbadc072a2e4f1e9"} Oct 29 22:42:23.070412 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.070345 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-9qhv9" event={"ID":"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8","Type":"ContainerStarted","Data":"b12aec2fedb8f3264afa0c6e4e36f3cc62676554e9adc0e336775257632dcf72"} Oct 29 22:42:23.072080 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.072009 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-868dt" event={"ID":"8f6c01b9-e044-413c-b533-09033d04f29e","Type":"ContainerStarted","Data":"b0e0c2bbab3f446ae654fa789a3fdca7a20acb188bbae830f145505f196b75fc"} Oct 29 22:42:23.073576 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.073443 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" event={"ID":"093ceb90-294f-4133-83a6-32548405d530","Type":"ContainerStarted","Data":"ac9dd7d8666ac55fac7d0ebe766e087035079f15b2f3dcffcfb734b1f0597a10"} Oct 29 22:42:23.082053 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.081138 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fh7rx" event={"ID":"a1a232f4-f67c-404f-a1f7-af0c3cc30180","Type":"ContainerStarted","Data":"78ba58a42087aa1823cbfb423afa93c3205120ef0ba622412f14ac3a2a2e1a6c"} Oct 29 22:42:23.082053 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.081173 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fh7rx" event={"ID":"a1a232f4-f67c-404f-a1f7-af0c3cc30180","Type":"ContainerStarted","Data":"9870c7f5e182878c16f0d6c428a1c62d407777effb29d920c0ffe042b0ed2873"} Oct 29 22:42:23.101049 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.099013 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-7hz4v" podStartSLOduration=963.098998231 podStartE2EDuration="16m3.098998231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:23.098663349 +0000 UTC m=+29.837797443" watchObservedRunningTime="2026-10-29 22:42:23.098998231 +0000 UTC m=+29.838132340" Oct 29 22:42:23.149413 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.148102 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-fh7rx" podStartSLOduration=963.148085024 podStartE2EDuration="16m3.148085024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:23.147274741 +0000 UTC m=+29.886408838" watchObservedRunningTime="2026-10-29 22:42:23.148085024 +0000 UTC m=+29.887219122" Oct 29 22:42:23.196469 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.195444 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-k8xnq" podStartSLOduration=963.195426047 podStartE2EDuration="16m3.195426047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:23.194485495 +0000 UTC m=+29.933619595" watchObservedRunningTime="2026-10-29 22:42:23.195426047 +0000 UTC m=+29.934560144" Oct 29 22:42:23.349280 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.347559 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8ncgh\" (UID: \"96726184-9be2-4685-93d1-415bbb8b551c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:23.349280 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.347654 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:23.349280 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.347691 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:23.349280 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.347741 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99577b75-29cd-430b-b5ac-2af3828586d0-service-ca-bundle\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:23.349280 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.347770 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:23.349280 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.347800 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/575ee3c4-7c77-4b31-9961-eebc60254d19-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rwfmh\" (UID: \"575ee3c4-7c77-4b31-9961-eebc60254d19\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rwfmh" Oct 29 22:42:23.349280 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.347833 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-metrics-certs\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:23.349280 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:23.347878 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:23.349280 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:23.348043 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:42:23.349280 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:23.348058 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f4dff4c9-87jff: secret "image-registry-tls" not found Oct 29 22:42:23.349280 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:23.348114 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls podName:884370ff-c54f-4d6b-af78-5c0813bb48eb nodeName:}" failed. No retries permitted until 2026-10-29 22:42:25.348097062 +0000 UTC m=+32.087231150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls") pod "image-registry-78f4dff4c9-87jff" (UID: "884370ff-c54f-4d6b-af78-5c0813bb48eb") : secret "image-registry-tls" not found Oct 29 22:42:23.349280 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:23.348541 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 22:42:23.349280 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:23.348597 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-tls podName:96726184-9be2-4685-93d1-415bbb8b551c nodeName:}" failed. No retries permitted until 2026-10-29 22:42:25.34858286 +0000 UTC m=+32.087716939 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8ncgh" (UID: "96726184-9be2-4685-93d1-415bbb8b551c") : secret "samples-operator-tls" not found Oct 29 22:42:23.349280 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:23.348666 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:42:23.349280 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:23.348676 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf56cb474-gx24x: secret "image-registry-tls" not found Oct 29 22:42:23.349280 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:23.348708 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls podName:877326a8-91a3-499f-8b37-4f80dfb00352 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:25.348697239 +0000 UTC m=+32.087831318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls") pod "image-registry-5cf56cb474-gx24x" (UID: "877326a8-91a3-499f-8b37-4f80dfb00352") : secret "image-registry-tls" not found Oct 29 22:42:23.349280 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:23.348760 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:42:23.350201 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:23.348805 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls podName:ad00084a-5a9f-489b-9122-2316f617f3f1 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:25.348793509 +0000 UTC m=+32.087927588 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dzx6b" (UID: "ad00084a-5a9f-489b-9122-2316f617f3f1") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:42:23.350201 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:23.348870 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99577b75-29cd-430b-b5ac-2af3828586d0-service-ca-bundle podName:99577b75-29cd-430b-b5ac-2af3828586d0 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:25.348860364 +0000 UTC m=+32.087994443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99577b75-29cd-430b-b5ac-2af3828586d0-service-ca-bundle") pod "router-default-7945df5b5-pcgp6" (UID: "99577b75-29cd-430b-b5ac-2af3828586d0") : configmap references non-existent config key: service-ca.crt Oct 29 22:42:23.350201 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:23.348921 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 22:42:23.350201 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:23.348949 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls podName:266bff54-6027-4822-91ce-be5cbd316ddc nodeName:}" failed. No retries permitted until 2026-10-29 22:42:25.348940432 +0000 UTC m=+32.088074510 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls") pod "dns-default-gj6vs" (UID: "266bff54-6027-4822-91ce-be5cbd316ddc") : secret "dns-default-metrics-tls" not found Oct 29 22:42:23.350201 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:23.349001 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:42:23.350201 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:23.349030 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/575ee3c4-7c77-4b31-9961-eebc60254d19-networking-console-plugin-cert podName:575ee3c4-7c77-4b31-9961-eebc60254d19 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:25.349021782 +0000 UTC m=+32.088155859 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/575ee3c4-7c77-4b31-9961-eebc60254d19-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-rwfmh" (UID: "575ee3c4-7c77-4b31-9961-eebc60254d19") : secret "networking-console-plugin-cert" not found Oct 29 22:42:23.350201 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:23.349082 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 22:42:23.350201 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:23.349111 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-metrics-certs podName:99577b75-29cd-430b-b5ac-2af3828586d0 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:25.349101632 +0000 UTC m=+32.088235710 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-metrics-certs") pod "router-default-7945df5b5-pcgp6" (UID: "99577b75-29cd-430b-b5ac-2af3828586d0") : secret "router-metrics-certs-default" not found Oct 29 22:42:25.094027 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:25.093220 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-868dt" event={"ID":"8f6c01b9-e044-413c-b533-09033d04f29e","Type":"ContainerStarted","Data":"d5554d881436a7fd9be8495f29f4f1d0c2263956c2398c77ca6aa0961860f919"} Oct 29 22:42:25.142200 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:25.141233 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-868dt" podStartSLOduration=919.901949143 podStartE2EDuration="15m22.141210601s" totalImagesPullingTime="2.239261458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:25.141083366 +0000 UTC m=+31.880217464" watchObservedRunningTime="2026-10-29 22:42:25.141210601 +0000 UTC m=+31.880344698" Oct 29 22:42:25.373886 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:25.372264 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8ncgh\" (UID: \"96726184-9be2-4685-93d1-415bbb8b551c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:25.373886 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:25.372361 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:25.373886 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:25.372397 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:25.373886 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:25.372442 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99577b75-29cd-430b-b5ac-2af3828586d0-service-ca-bundle\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:25.373886 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:25.372472 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:25.373886 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:25.372504 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/575ee3c4-7c77-4b31-9961-eebc60254d19-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rwfmh\" (UID: \"575ee3c4-7c77-4b31-9961-eebc60254d19\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rwfmh" Oct 29 22:42:25.373886 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:25.372534 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-metrics-certs\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:25.373886 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:25.372578 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:25.373886 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:25.372747 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:42:25.373886 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:25.372763 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f4dff4c9-87jff: secret "image-registry-tls" not found Oct 29 22:42:25.373886 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:25.372825 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls podName:884370ff-c54f-4d6b-af78-5c0813bb48eb nodeName:}" failed. No retries permitted until 2026-10-29 22:42:29.372804129 +0000 UTC m=+36.111938229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls") pod "image-registry-78f4dff4c9-87jff" (UID: "884370ff-c54f-4d6b-af78-5c0813bb48eb") : secret "image-registry-tls" not found Oct 29 22:42:25.373886 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:25.373257 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 22:42:25.373886 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:25.373318 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-tls podName:96726184-9be2-4685-93d1-415bbb8b551c nodeName:}" failed. No retries permitted until 2026-10-29 22:42:29.373300463 +0000 UTC m=+36.112434555 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8ncgh" (UID: "96726184-9be2-4685-93d1-415bbb8b551c") : secret "samples-operator-tls" not found Oct 29 22:42:25.373886 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:25.373387 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:42:25.373886 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:25.373398 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf56cb474-gx24x: secret "image-registry-tls" not found Oct 29 22:42:25.373886 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:25.373432 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls podName:877326a8-91a3-499f-8b37-4f80dfb00352 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:29.3734208 +0000 UTC m=+36.112554889 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls") pod "image-registry-5cf56cb474-gx24x" (UID: "877326a8-91a3-499f-8b37-4f80dfb00352") : secret "image-registry-tls" not found Oct 29 22:42:25.373886 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:25.373483 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:42:25.374821 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:25.373512 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls podName:ad00084a-5a9f-489b-9122-2316f617f3f1 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:29.373503253 +0000 UTC m=+36.112637342 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dzx6b" (UID: "ad00084a-5a9f-489b-9122-2316f617f3f1") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:42:25.374821 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:25.373568 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99577b75-29cd-430b-b5ac-2af3828586d0-service-ca-bundle podName:99577b75-29cd-430b-b5ac-2af3828586d0 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:29.373559077 +0000 UTC m=+36.112693166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99577b75-29cd-430b-b5ac-2af3828586d0-service-ca-bundle") pod "router-default-7945df5b5-pcgp6" (UID: "99577b75-29cd-430b-b5ac-2af3828586d0") : configmap references non-existent config key: service-ca.crt Oct 29 22:42:25.374821 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:25.373618 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 22:42:25.374821 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:25.373648 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls podName:266bff54-6027-4822-91ce-be5cbd316ddc nodeName:}" failed. No retries permitted until 2026-10-29 22:42:29.373639228 +0000 UTC m=+36.112773318 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls") pod "dns-default-gj6vs" (UID: "266bff54-6027-4822-91ce-be5cbd316ddc") : secret "dns-default-metrics-tls" not found Oct 29 22:42:25.374821 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:25.373703 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:42:25.374821 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:25.373732 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/575ee3c4-7c77-4b31-9961-eebc60254d19-networking-console-plugin-cert podName:575ee3c4-7c77-4b31-9961-eebc60254d19 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:29.373723277 +0000 UTC m=+36.112857366 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/575ee3c4-7c77-4b31-9961-eebc60254d19-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-rwfmh" (UID: "575ee3c4-7c77-4b31-9961-eebc60254d19") : secret "networking-console-plugin-cert" not found Oct 29 22:42:25.374821 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:25.373770 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 22:42:25.374821 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:25.373796 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-metrics-certs podName:99577b75-29cd-430b-b5ac-2af3828586d0 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:29.373787122 +0000 UTC m=+36.112921211 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-metrics-certs") pod "router-default-7945df5b5-pcgp6" (UID: "99577b75-29cd-430b-b5ac-2af3828586d0") : secret "router-metrics-certs-default" not found Oct 29 22:42:27.101419 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.101385 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-9qhv9" event={"ID":"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8","Type":"ContainerStarted","Data":"eeb8ffb67da2a37b6680bb72557c01fd54a05a5a566c9f8a1c75f2786c0dd12c"} Oct 29 22:42:27.104458 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.103998 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" event={"ID":"093ceb90-294f-4133-83a6-32548405d530","Type":"ContainerStarted","Data":"45366dc34188d8f753af295ef018b066b8b533816d92e6b514a65863864e0d55"} Oct 29 22:42:27.106360 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.106338 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" event={"ID":"a109715c-c29a-48d1-a573-3e7800f48259","Type":"ContainerStarted","Data":"f4bf63c72257935d12452566cc01f3a0896e3340b4d39913d0ed28840ada515d"} Oct 29 22:42:27.107987 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.107966 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/0.log" Oct 29 22:42:27.108074 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.107997 2581 generic.go:356] "Generic (PLEG): container finished" podID="9e91a303-9b80-47f5-a502-fbb6365eaa54" containerID="751300d0b0d491744ef468ceec0c149c21ee02725dead2b72d1765d24d9788da" exitCode=255 Oct 29 22:42:27.108074 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.108028 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" event={"ID":"9e91a303-9b80-47f5-a502-fbb6365eaa54","Type":"ContainerDied","Data":"751300d0b0d491744ef468ceec0c149c21ee02725dead2b72d1765d24d9788da"} Oct 29 22:42:27.108400 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.108305 2581 scope.go:122] "RemoveContainer" containerID="751300d0b0d491744ef468ceec0c149c21ee02725dead2b72d1765d24d9788da" Oct 29 22:42:27.168106 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.168019 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-9qhv9" podStartSLOduration=920.054019885 podStartE2EDuration="15m24.167990844s" totalImagesPullingTime="4.113970959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:27.167120921 +0000 UTC m=+33.906255021" watchObservedRunningTime="2026-10-29 22:42:27.167990844 +0000 UTC m=+33.907124941" Oct 29 22:42:27.271008 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.270942 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" podStartSLOduration=920.002310232 podStartE2EDuration="15m24.270925837s" totalImagesPullingTime="4.268615605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:27.270770785 +0000 UTC m=+34.009904882" watchObservedRunningTime="2026-10-29 22:42:27.270925837 +0000 UTC m=+34.010059939" Oct 29 22:42:27.306538 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.306503 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6rqj6"] Oct 29 22:42:27.330230 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.330113 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:27.372261 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.372212 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 22:42:27.372261 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.372250 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nxz4d\"" Oct 29 22:42:27.373812 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.373789 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 22:42:27.373940 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.373824 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" podStartSLOduration=919.903602089 podStartE2EDuration="15m24.373809136s" totalImagesPullingTime="4.470207047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:27.370328624 +0000 UTC m=+34.109462720" watchObservedRunningTime="2026-10-29 22:42:27.373809136 +0000 UTC m=+34.112943232" Oct 29 22:42:27.375499 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.375478 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6rqj6"] Oct 29 22:42:27.430145 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.430115 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-vqgrg-xtvww"] Oct 29 22:42:27.443125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.443101 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:27.454105 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.454078 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vqgrg-xtvww"] Oct 29 22:42:27.497683 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.497647 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs\") pod \"network-metrics-daemon-mxg8q\" (UID: \"286133c3-abe5-4954-83a1-8665c1785b92\") " pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:27.497874 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.497785 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:27.497874 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:27.497818 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 29 22:42:27.497874 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.497818 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:27.498034 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.497893 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-crio-socket\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:27.498034 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:27.497913 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs podName:286133c3-abe5-4954-83a1-8665c1785b92 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:59.497896146 +0000 UTC m=+66.237030221 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs") pod "network-metrics-daemon-mxg8q" (UID: "286133c3-abe5-4954-83a1-8665c1785b92") : secret "metrics-daemon-secret" not found Oct 29 22:42:27.498034 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.497991 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:27.498034 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.498014 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrhns\" (UniqueName: \"kubernetes.io/projected/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-api-access-jrhns\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:27.498216 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.498038 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-data-volume\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:27.501805 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.501778 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:27.590401 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.590321 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:27.598776 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.598742 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:27.598915 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.598782 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:27.598915 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.598804 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-crio-socket\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:27.598915 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.598837 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29d1f02-5697-4d94-8a21-0a39e8f473ed-service-ca-bundle\") pod \"periodic-gathering-vqgrg-xtvww\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:27.598915 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.598880 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk8rz\" (UniqueName: \"kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz\") pod \"network-check-target-j8fft\" (UID: \"463bc340-b1dc-41e6-9203-fcd2672f8a17\") " pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:27.598915 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:27.598879 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 22:42:27.598915 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:27.598910 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:42:27.599169 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:27.598959 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls podName:d518d30f-c46b-49ca-b74c-9f4cf0ca71e9 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:28.098948229 +0000 UTC m=+34.838082305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6rqj6" (UID: "d518d30f-c46b-49ca-b74c-9f4cf0ca71e9") : secret "insights-runtime-extractor-tls" not found Oct 29 22:42:27.599169 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:27.599020 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm podName:d518d30f-c46b-49ca-b74c-9f4cf0ca71e9 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:28.098984672 +0000 UTC m=+34.838118755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6rqj6" (UID: "d518d30f-c46b-49ca-b74c-9f4cf0ca71e9") : configmap "kube-rbac-proxy" not found Oct 29 22:42:27.599169 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.599019 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-crio-socket\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:27.599169 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.599077 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrhns\" (UniqueName: \"kubernetes.io/projected/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-api-access-jrhns\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:27.599169 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.599099 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29d1f02-5697-4d94-8a21-0a39e8f473ed-trusted-ca-bundle\") pod \"periodic-gathering-vqgrg-xtvww\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:27.599169 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.599163 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c29d1f02-5697-4d94-8a21-0a39e8f473ed-archives-path\") pod \"periodic-gathering-vqgrg-xtvww\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:27.599467 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.599265 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-data-volume\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:27.599467 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.599430 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx7m8\" (UniqueName: \"kubernetes.io/projected/c29d1f02-5697-4d94-8a21-0a39e8f473ed-kube-api-access-cx7m8\") pod \"periodic-gathering-vqgrg-xtvww\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:27.599568 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.599548 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-data-volume\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:27.601651 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.601622 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk8rz\" (UniqueName: \"kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz\") pod \"network-check-target-j8fft\" (UID: \"463bc340-b1dc-41e6-9203-fcd2672f8a17\") " pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:27.636460 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.636428 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrhns\" (UniqueName: \"kubernetes.io/projected/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-api-access-jrhns\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:27.700809 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.700776 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29d1f02-5697-4d94-8a21-0a39e8f473ed-trusted-ca-bundle\") pod \"periodic-gathering-vqgrg-xtvww\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:27.700908 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.700825 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c29d1f02-5697-4d94-8a21-0a39e8f473ed-archives-path\") pod \"periodic-gathering-vqgrg-xtvww\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:27.700965 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.700947 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx7m8\" (UniqueName: \"kubernetes.io/projected/c29d1f02-5697-4d94-8a21-0a39e8f473ed-kube-api-access-cx7m8\") pod \"periodic-gathering-vqgrg-xtvww\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:27.701228 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.701136 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29d1f02-5697-4d94-8a21-0a39e8f473ed-service-ca-bundle\") pod \"periodic-gathering-vqgrg-xtvww\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:27.701827 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.701779 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c29d1f02-5697-4d94-8a21-0a39e8f473ed-archives-path\") pod \"periodic-gathering-vqgrg-xtvww\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:27.702125 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.702072 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29d1f02-5697-4d94-8a21-0a39e8f473ed-service-ca-bundle\") pod \"periodic-gathering-vqgrg-xtvww\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:27.702535 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.702511 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29d1f02-5697-4d94-8a21-0a39e8f473ed-trusted-ca-bundle\") pod \"periodic-gathering-vqgrg-xtvww\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:27.711264 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.711233 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zl99j"] Oct 29 22:42:27.740320 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.740263 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx7m8\" (UniqueName: \"kubernetes.io/projected/c29d1f02-5697-4d94-8a21-0a39e8f473ed-kube-api-access-cx7m8\") pod \"periodic-gathering-vqgrg-xtvww\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:27.749212 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.748690 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n56zr"] Oct 29 22:42:27.749212 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.749205 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zl99j" Oct 29 22:42:27.767227 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.766821 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n56zr" Oct 29 22:42:27.768322 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.768301 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:27.771401 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.771382 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zl99j"] Oct 29 22:42:27.771466 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.771429 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n56zr"] Oct 29 22:42:27.813785 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.813752 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-rzm2m\"" Oct 29 22:42:27.834154 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.834014 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 29 22:42:27.834154 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.834119 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 29 22:42:27.887713 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.887361 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:27.903453 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.903419 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5d85\" (UniqueName: \"kubernetes.io/projected/9483bf09-4676-47db-a279-84aabe9d17ed-kube-api-access-w5d85\") pod \"migrator-66df5d496b-zl99j\" (UID: \"9483bf09-4676-47db-a279-84aabe9d17ed\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zl99j" Oct 29 22:42:27.903607 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.903474 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4z7h\" (UniqueName: \"kubernetes.io/projected/ce5dd070-fce7-44da-aaeb-017109fc98c9-kube-api-access-h4z7h\") pod \"migrator-66df5d496b-n56zr\" (UID: \"ce5dd070-fce7-44da-aaeb-017109fc98c9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n56zr" Oct 29 22:42:27.973122 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:27.973094 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b2cf2"] Oct 29 22:42:27.974497 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:42:27.974449 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ea529e_ab5d_419c_87bf_fe0bd1a6460f.slice/crio-3e0e85234849cda82191e957089de80d763789a100521253717a21405e99c869 WatchSource:0}: Error finding container 3e0e85234849cda82191e957089de80d763789a100521253717a21405e99c869: Status 404 returned error can't find the container with id 3e0e85234849cda82191e957089de80d763789a100521253717a21405e99c869 Oct 29 22:42:28.004819 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.004787 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5d85\" (UniqueName: \"kubernetes.io/projected/9483bf09-4676-47db-a279-84aabe9d17ed-kube-api-access-w5d85\") pod \"migrator-66df5d496b-zl99j\" (UID: \"9483bf09-4676-47db-a279-84aabe9d17ed\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zl99j" Oct 29 22:42:28.004961 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.004856 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4z7h\" (UniqueName: \"kubernetes.io/projected/ce5dd070-fce7-44da-aaeb-017109fc98c9-kube-api-access-h4z7h\") pod \"migrator-66df5d496b-n56zr\" (UID: \"ce5dd070-fce7-44da-aaeb-017109fc98c9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n56zr" Oct 29 22:42:28.030421 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.030393 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4z7h\" (UniqueName: \"kubernetes.io/projected/ce5dd070-fce7-44da-aaeb-017109fc98c9-kube-api-access-h4z7h\") pod \"migrator-66df5d496b-n56zr\" (UID: \"ce5dd070-fce7-44da-aaeb-017109fc98c9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n56zr" Oct 29 22:42:28.031437 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.031414 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5d85\" (UniqueName: \"kubernetes.io/projected/9483bf09-4676-47db-a279-84aabe9d17ed-kube-api-access-w5d85\") pod \"migrator-66df5d496b-zl99j\" (UID: \"9483bf09-4676-47db-a279-84aabe9d17ed\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zl99j" Oct 29 22:42:28.077090 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.077034 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vqgrg-xtvww"] Oct 29 22:42:28.079168 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:42:28.079127 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29d1f02_5697_4d94_8a21_0a39e8f473ed.slice/crio-9f59a804858118aa038ff14cb50db8dcd52f6a3704bb85fbcf18447550d056fe WatchSource:0}: Error finding container 9f59a804858118aa038ff14cb50db8dcd52f6a3704bb85fbcf18447550d056fe: Status 404 returned error can't find the container with id 9f59a804858118aa038ff14cb50db8dcd52f6a3704bb85fbcf18447550d056fe Oct 29 22:42:28.086976 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.086958 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zl99j" Oct 29 22:42:28.105475 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.105453 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:28.105762 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.105592 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:28.105762 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:28.105617 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:42:28.105762 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:28.105657 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 22:42:28.105762 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:28.105703 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls podName:d518d30f-c46b-49ca-b74c-9f4cf0ca71e9 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:29.105681586 +0000 UTC m=+35.844815673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6rqj6" (UID: "d518d30f-c46b-49ca-b74c-9f4cf0ca71e9") : secret "insights-runtime-extractor-tls" not found Oct 29 22:42:28.105762 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:28.105723 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm podName:d518d30f-c46b-49ca-b74c-9f4cf0ca71e9 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:29.105713249 +0000 UTC m=+35.844847326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6rqj6" (UID: "d518d30f-c46b-49ca-b74c-9f4cf0ca71e9") : configmap "kube-rbac-proxy" not found Oct 29 22:42:28.112578 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.112553 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vqgrg-xtvww" event={"ID":"c29d1f02-5697-4d94-8a21-0a39e8f473ed","Type":"ContainerStarted","Data":"9f59a804858118aa038ff14cb50db8dcd52f6a3704bb85fbcf18447550d056fe"} Oct 29 22:42:28.113687 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.113667 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b2cf2" event={"ID":"41ea529e-ab5d-419c-87bf-fe0bd1a6460f","Type":"ContainerStarted","Data":"3e0e85234849cda82191e957089de80d763789a100521253717a21405e99c869"} Oct 29 22:42:28.115388 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.115366 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qbv9d" event={"ID":"d6f91740-e20b-439e-be7d-4b444f4835fa","Type":"ContainerStarted","Data":"377ed02d8f8cea96491daa8aa58fa61c637db0540914d13222b0479275ddfe5f"} Oct 29 22:42:28.116148 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.116132 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n56zr" Oct 29 22:42:28.117221 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.117181 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 22:42:28.117675 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.117650 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/0.log" Oct 29 22:42:28.117746 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.117692 2581 generic.go:356] "Generic (PLEG): container finished" podID="9e91a303-9b80-47f5-a502-fbb6365eaa54" containerID="f0fcd8c76c7857be846eac042de3c1bf7b719752c31262e9fa639c35cfdc6aac" exitCode=255 Oct 29 22:42:28.117797 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.117771 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" event={"ID":"9e91a303-9b80-47f5-a502-fbb6365eaa54","Type":"ContainerDied","Data":"f0fcd8c76c7857be846eac042de3c1bf7b719752c31262e9fa639c35cfdc6aac"} Oct 29 22:42:28.117851 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.117824 2581 scope.go:122] "RemoveContainer" containerID="751300d0b0d491744ef468ceec0c149c21ee02725dead2b72d1765d24d9788da" Oct 29 22:42:28.118249 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.118233 2581 scope.go:122] "RemoveContainer" containerID="f0fcd8c76c7857be846eac042de3c1bf7b719752c31262e9fa639c35cfdc6aac" Oct 29 22:42:28.118432 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:28.118409 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-lm55w_openshift-console-operator(9e91a303-9b80-47f5-a502-fbb6365eaa54)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" podUID="9e91a303-9b80-47f5-a502-fbb6365eaa54" Oct 29 22:42:28.119490 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.119466 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kw5c9" event={"ID":"85ab246f-f679-40c5-a127-c0fd5a4a621e","Type":"ContainerStarted","Data":"3e204ade1ff2bb517c845aa63ac70e3663559ea754dc90a47363217195100d19"} Oct 29 22:42:28.174795 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.174726 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qbv9d" podStartSLOduration=2.487520859 podStartE2EDuration="7.174705946s" totalImagesPullingTime="4.687185087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:28.170858659 +0000 UTC m=+34.909992755" watchObservedRunningTime="2026-10-29 22:42:28.174705946 +0000 UTC m=+34.913840037" Oct 29 22:42:28.201726 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.201666 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j8fft"] Oct 29 22:42:28.207012 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:42:28.206716 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463bc340_b1dc_41e6_9203_fcd2672f8a17.slice/crio-710f6482706cf21d13a0be40293e5c922dc9549d6d94c8e010c661f794105f8c WatchSource:0}: Error finding container 710f6482706cf21d13a0be40293e5c922dc9549d6d94c8e010c661f794105f8c: Status 404 returned error can't find the container with id 710f6482706cf21d13a0be40293e5c922dc9549d6d94c8e010c661f794105f8c Oct 29 22:42:28.260763 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.260737 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-d49cl"] Oct 29 22:42:28.278980 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.278935 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kw5c9" podStartSLOduration=919.995845051 podStartE2EDuration="15m25.278922233s" totalImagesPullingTime="5.283077182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:28.277131911 +0000 UTC m=+35.016266007" watchObservedRunningTime="2026-10-29 22:42:28.278922233 +0000 UTC m=+35.018056319" Oct 29 22:42:28.290942 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.290906 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-d49cl"] Oct 29 22:42:28.291327 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.291248 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-d49cl" Oct 29 22:42:28.316843 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.316814 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 29 22:42:28.316843 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.316830 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 29 22:42:28.316843 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.316831 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 29 22:42:28.317035 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.316889 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 29 22:42:28.317035 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.316928 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-n2tqz\"" Oct 29 22:42:28.410089 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.410008 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/73b41114-4ff0-4357-98b5-a871335ee3de-signing-key\") pod \"service-ca-5fffc97bdf-d49cl\" (UID: \"73b41114-4ff0-4357-98b5-a871335ee3de\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d49cl" Oct 29 22:42:28.410269 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.410204 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/73b41114-4ff0-4357-98b5-a871335ee3de-signing-cabundle\") pod \"service-ca-5fffc97bdf-d49cl\" (UID: \"73b41114-4ff0-4357-98b5-a871335ee3de\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d49cl" Oct 29 22:42:28.410269 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.410243 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfslj\" (UniqueName: \"kubernetes.io/projected/73b41114-4ff0-4357-98b5-a871335ee3de-kube-api-access-vfslj\") pod \"service-ca-5fffc97bdf-d49cl\" (UID: \"73b41114-4ff0-4357-98b5-a871335ee3de\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d49cl" Oct 29 22:42:28.416218 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.416176 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zl99j"] Oct 29 22:42:28.418174 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:42:28.418142 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9483bf09_4676_47db_a279_84aabe9d17ed.slice/crio-99aaddab36725a8f6ffbbf5d65006c25fbd7bcca3de186b40c5774dbd8a614cb WatchSource:0}: Error finding container 99aaddab36725a8f6ffbbf5d65006c25fbd7bcca3de186b40c5774dbd8a614cb: Status 404 returned error can't find the container with id 99aaddab36725a8f6ffbbf5d65006c25fbd7bcca3de186b40c5774dbd8a614cb Oct 29 22:42:28.492548 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.492515 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n56zr"] Oct 29 22:42:28.494417 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:42:28.494391 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce5dd070_fce7_44da_aaeb_017109fc98c9.slice/crio-74d538b607b308700c2ddf7b631b7a4f89045fef1002029b6d323b160552ac49 WatchSource:0}: Error finding container 74d538b607b308700c2ddf7b631b7a4f89045fef1002029b6d323b160552ac49: Status 404 returned error can't find the container with id 74d538b607b308700c2ddf7b631b7a4f89045fef1002029b6d323b160552ac49 Oct 29 22:42:28.511234 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.511180 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/73b41114-4ff0-4357-98b5-a871335ee3de-signing-key\") pod \"service-ca-5fffc97bdf-d49cl\" (UID: \"73b41114-4ff0-4357-98b5-a871335ee3de\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d49cl" Oct 29 22:42:28.511367 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.511278 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/73b41114-4ff0-4357-98b5-a871335ee3de-signing-cabundle\") pod \"service-ca-5fffc97bdf-d49cl\" (UID: \"73b41114-4ff0-4357-98b5-a871335ee3de\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d49cl" Oct 29 22:42:28.511367 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.511303 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfslj\" (UniqueName: \"kubernetes.io/projected/73b41114-4ff0-4357-98b5-a871335ee3de-kube-api-access-vfslj\") pod \"service-ca-5fffc97bdf-d49cl\" (UID: \"73b41114-4ff0-4357-98b5-a871335ee3de\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d49cl" Oct 29 22:42:28.512044 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.512017 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/73b41114-4ff0-4357-98b5-a871335ee3de-signing-cabundle\") pod \"service-ca-5fffc97bdf-d49cl\" (UID: \"73b41114-4ff0-4357-98b5-a871335ee3de\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d49cl" Oct 29 22:42:28.513809 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.513773 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/73b41114-4ff0-4357-98b5-a871335ee3de-signing-key\") pod \"service-ca-5fffc97bdf-d49cl\" (UID: \"73b41114-4ff0-4357-98b5-a871335ee3de\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d49cl" Oct 29 22:42:28.537529 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.537491 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfslj\" (UniqueName: \"kubernetes.io/projected/73b41114-4ff0-4357-98b5-a871335ee3de-kube-api-access-vfslj\") pod \"service-ca-5fffc97bdf-d49cl\" (UID: \"73b41114-4ff0-4357-98b5-a871335ee3de\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d49cl" Oct 29 22:42:28.616376 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.616348 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-d49cl" Oct 29 22:42:28.954466 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:28.954437 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-d49cl"] Oct 29 22:42:28.962332 ip-10-0-135-80 kubenswrapper[2581]: W1029 22:42:28.962300 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b41114_4ff0_4357_98b5_a871335ee3de.slice/crio-a37a3da87cc440a67f1cefaefb01ac48e701dfc430b6bce0687dd5782f6833d7 WatchSource:0}: Error finding container a37a3da87cc440a67f1cefaefb01ac48e701dfc430b6bce0687dd5782f6833d7: Status 404 returned error can't find the container with id a37a3da87cc440a67f1cefaefb01ac48e701dfc430b6bce0687dd5782f6833d7 Oct 29 22:42:29.118101 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.117729 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:29.118101 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.117784 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:29.118101 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.117879 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 22:42:29.118101 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.117955 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm podName:d518d30f-c46b-49ca-b74c-9f4cf0ca71e9 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:31.117933634 +0000 UTC m=+37.857067713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6rqj6" (UID: "d518d30f-c46b-49ca-b74c-9f4cf0ca71e9") : configmap "kube-rbac-proxy" not found Oct 29 22:42:29.118101 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.118015 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:42:29.118101 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.118071 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls podName:d518d30f-c46b-49ca-b74c-9f4cf0ca71e9 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:31.118054796 +0000 UTC m=+37.857188886 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6rqj6" (UID: "d518d30f-c46b-49ca-b74c-9f4cf0ca71e9") : secret "insights-runtime-extractor-tls" not found Oct 29 22:42:29.125984 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.125360 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-d49cl" event={"ID":"73b41114-4ff0-4357-98b5-a871335ee3de","Type":"ContainerStarted","Data":"b22b23e590a873f1a4bcf9f7f49eb73312a0ee6f5f8c1d8a18b1311189ec8579"} Oct 29 22:42:29.125984 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.125422 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-d49cl" event={"ID":"73b41114-4ff0-4357-98b5-a871335ee3de","Type":"ContainerStarted","Data":"a37a3da87cc440a67f1cefaefb01ac48e701dfc430b6bce0687dd5782f6833d7"} Oct 29 22:42:29.128069 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.128041 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j8fft" event={"ID":"463bc340-b1dc-41e6-9203-fcd2672f8a17","Type":"ContainerStarted","Data":"90235302b4f4a06dcebd3eb2e67a66fd5fe71f18164971982ebde104578292be"} Oct 29 22:42:29.128610 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.128220 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j8fft" event={"ID":"463bc340-b1dc-41e6-9203-fcd2672f8a17","Type":"ContainerStarted","Data":"710f6482706cf21d13a0be40293e5c922dc9549d6d94c8e010c661f794105f8c"} Oct 29 22:42:29.128610 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.128578 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:29.130649 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.130499 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vqgrg-xtvww" event={"ID":"c29d1f02-5697-4d94-8a21-0a39e8f473ed","Type":"ContainerStarted","Data":"6e1c2ee96f549cfeeb4d76c428b0c5bee69900b252535e6874b48c165053a95e"} Oct 29 22:42:29.133722 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.133704 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 22:42:29.134040 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.134016 2581 scope.go:122] "RemoveContainer" containerID="f0fcd8c76c7857be846eac042de3c1bf7b719752c31262e9fa639c35cfdc6aac" Oct 29 22:42:29.134218 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.134178 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-lm55w_openshift-console-operator(9e91a303-9b80-47f5-a502-fbb6365eaa54)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" podUID="9e91a303-9b80-47f5-a502-fbb6365eaa54" Oct 29 22:42:29.136634 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.136570 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n56zr" event={"ID":"ce5dd070-fce7-44da-aaeb-017109fc98c9","Type":"ContainerStarted","Data":"74d538b607b308700c2ddf7b631b7a4f89045fef1002029b6d323b160552ac49"} Oct 29 22:42:29.138094 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.138018 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zl99j" event={"ID":"9483bf09-4676-47db-a279-84aabe9d17ed","Type":"ContainerStarted","Data":"99aaddab36725a8f6ffbbf5d65006c25fbd7bcca3de186b40c5774dbd8a614cb"} Oct 29 22:42:29.173211 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.173130 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-d49cl" podStartSLOduration=1.1731116 podStartE2EDuration="1.1731116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:29.171373377 +0000 UTC m=+35.910507473" watchObservedRunningTime="2026-10-29 22:42:29.1731116 +0000 UTC m=+35.912245696" Oct 29 22:42:29.274727 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.273478 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-vqgrg-xtvww" podStartSLOduration=2.273457538 podStartE2EDuration="2.273457538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:29.27299008 +0000 UTC m=+36.012124176" watchObservedRunningTime="2026-10-29 22:42:29.273457538 +0000 UTC m=+36.012591634" Oct 29 22:42:29.321712 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.321655 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j8fft" podStartSLOduration=36.321641736 podStartE2EDuration="36.321641736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:29.321395221 +0000 UTC m=+36.060529318" watchObservedRunningTime="2026-10-29 22:42:29.321641736 +0000 UTC m=+36.060775823" Oct 29 22:42:29.421495 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.421456 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8ncgh\" (UID: \"96726184-9be2-4685-93d1-415bbb8b551c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:29.421667 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.421544 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:29.421667 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.421579 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:29.421667 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.421627 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99577b75-29cd-430b-b5ac-2af3828586d0-service-ca-bundle\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:29.421667 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.421642 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 22:42:29.421667 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.421657 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:29.421925 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.421685 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/575ee3c4-7c77-4b31-9961-eebc60254d19-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rwfmh\" (UID: \"575ee3c4-7c77-4b31-9961-eebc60254d19\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rwfmh" Oct 29 22:42:29.421925 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.421702 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-tls podName:96726184-9be2-4685-93d1-415bbb8b551c nodeName:}" failed. No retries permitted until 2026-10-29 22:42:37.421686325 +0000 UTC m=+44.160820399 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8ncgh" (UID: "96726184-9be2-4685-93d1-415bbb8b551c") : secret "samples-operator-tls" not found Oct 29 22:42:29.421925 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.421733 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-metrics-certs\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:29.421925 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.421768 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:42:29.421925 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:29.421783 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:29.421925 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.421818 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/575ee3c4-7c77-4b31-9961-eebc60254d19-networking-console-plugin-cert podName:575ee3c4-7c77-4b31-9961-eebc60254d19 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:37.421804116 +0000 UTC m=+44.160938196 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/575ee3c4-7c77-4b31-9961-eebc60254d19-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-rwfmh" (UID: "575ee3c4-7c77-4b31-9961-eebc60254d19") : secret "networking-console-plugin-cert" not found Oct 29 22:42:29.421925 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.421860 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:42:29.421925 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.421868 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f4dff4c9-87jff: secret "image-registry-tls" not found Oct 29 22:42:29.421925 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.421887 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:42:29.421925 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.421891 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls podName:884370ff-c54f-4d6b-af78-5c0813bb48eb nodeName:}" failed. No retries permitted until 2026-10-29 22:42:37.421884944 +0000 UTC m=+44.161019018 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls") pod "image-registry-78f4dff4c9-87jff" (UID: "884370ff-c54f-4d6b-af78-5c0813bb48eb") : secret "image-registry-tls" not found Oct 29 22:42:29.421925 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.421898 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf56cb474-gx24x: secret "image-registry-tls" not found Oct 29 22:42:29.421925 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.421927 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls podName:877326a8-91a3-499f-8b37-4f80dfb00352 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:37.421917751 +0000 UTC m=+44.161051831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls") pod "image-registry-5cf56cb474-gx24x" (UID: "877326a8-91a3-499f-8b37-4f80dfb00352") : secret "image-registry-tls" not found Oct 29 22:42:29.422413 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.421955 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 22:42:29.422413 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.421977 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:42:29.422413 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.422007 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls podName:ad00084a-5a9f-489b-9122-2316f617f3f1 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:37.421997012 +0000 UTC m=+44.161131089 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dzx6b" (UID: "ad00084a-5a9f-489b-9122-2316f617f3f1") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:42:29.422413 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.422021 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-metrics-certs podName:99577b75-29cd-430b-b5ac-2af3828586d0 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:37.422012673 +0000 UTC m=+44.161146750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-metrics-certs") pod "router-default-7945df5b5-pcgp6" (UID: "99577b75-29cd-430b-b5ac-2af3828586d0") : secret "router-metrics-certs-default" not found Oct 29 22:42:29.422413 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.422054 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99577b75-29cd-430b-b5ac-2af3828586d0-service-ca-bundle podName:99577b75-29cd-430b-b5ac-2af3828586d0 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:37.422043354 +0000 UTC m=+44.161177427 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99577b75-29cd-430b-b5ac-2af3828586d0-service-ca-bundle") pod "router-default-7945df5b5-pcgp6" (UID: "99577b75-29cd-430b-b5ac-2af3828586d0") : configmap references non-existent config key: service-ca.crt Oct 29 22:42:29.422413 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.422065 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 22:42:29.422413 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:29.422091 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls podName:266bff54-6027-4822-91ce-be5cbd316ddc nodeName:}" failed. No retries permitted until 2026-10-29 22:42:37.422083157 +0000 UTC m=+44.161217251 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls") pod "dns-default-gj6vs" (UID: "266bff54-6027-4822-91ce-be5cbd316ddc") : secret "dns-default-metrics-tls" not found Oct 29 22:42:31.142230 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:31.142197 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:31.142780 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:31.142241 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:31.142780 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:31.142343 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 22:42:31.142780 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:31.142372 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:42:31.142780 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:31.142434 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm podName:d518d30f-c46b-49ca-b74c-9f4cf0ca71e9 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:35.142412606 +0000 UTC m=+41.881546684 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6rqj6" (UID: "d518d30f-c46b-49ca-b74c-9f4cf0ca71e9") : configmap "kube-rbac-proxy" not found Oct 29 22:42:31.142780 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:31.142458 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls podName:d518d30f-c46b-49ca-b74c-9f4cf0ca71e9 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:35.142448044 +0000 UTC m=+41.881582132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6rqj6" (UID: "d518d30f-c46b-49ca-b74c-9f4cf0ca71e9") : secret "insights-runtime-extractor-tls" not found Oct 29 22:42:31.150609 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:31.150578 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n56zr" event={"ID":"ce5dd070-fce7-44da-aaeb-017109fc98c9","Type":"ContainerStarted","Data":"5576e666ea84c94d5e4ed2c75b8cbf223df8abf8d72be986350babe3d3bee184"} Oct 29 22:42:31.153085 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:31.153043 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zl99j" event={"ID":"9483bf09-4676-47db-a279-84aabe9d17ed","Type":"ContainerStarted","Data":"d970039fdad128a86be56b179820fb9e1f770343c79e4f71975cc94e3ebfcd3e"} Oct 29 22:42:31.196238 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:31.196168 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n56zr" podStartSLOduration=2.493421057 podStartE2EDuration="4.196154894s" totalImagesPullingTime="1.702733837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:31.195570608 +0000 UTC m=+37.934704707" watchObservedRunningTime="2026-10-29 22:42:31.196154894 +0000 UTC m=+37.935288990" Oct 29 22:42:31.245887 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:31.245834 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zl99j" podStartSLOduration=2.4823692680000002 podStartE2EDuration="4.24581772s" totalImagesPullingTime="1.763448452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:31.244772107 +0000 UTC m=+37.983906202" watchObservedRunningTime="2026-10-29 22:42:31.24581772 +0000 UTC m=+37.984951816" Oct 29 22:42:32.012540 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:32.012499 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:32.012728 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:32.012575 2581 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:32.013015 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:32.012999 2581 scope.go:122] "RemoveContainer" containerID="f0fcd8c76c7857be846eac042de3c1bf7b719752c31262e9fa639c35cfdc6aac" Oct 29 22:42:32.013304 ip-10-0-135-80 kubenswrapper[2581]: E1029 22:42:32.013247 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-lm55w_openshift-console-operator(9e91a303-9b80-47f5-a502-fbb6365eaa54)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" podUID="9e91a303-9b80-47f5-a502-fbb6365eaa54" Oct 29 22:42:33.161684 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:33.161646 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b2cf2" event={"ID":"41ea529e-ab5d-419c-87bf-fe0bd1a6460f","Type":"ContainerStarted","Data":"751f625cf7de230c4c61cf985c54cbeae648545c50194b58497fc6f884d8185e"} Oct 29 22:42:33.188643 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:33.188614 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5czm4_af926a18-1147-4786-8bd9-8a6727abb6fa/dns-node-resolver/0.log" Oct 29 22:42:33.206630 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:33.206581 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b2cf2" podStartSLOduration=34.219326106 podStartE2EDuration="39.206567905s" totalImagesPullingTime="4.987241799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:33.206078638 +0000 UTC m=+39.945212734" watchObservedRunningTime="2026-10-29 22:42:33.206567905 +0000 UTC m=+39.945701994" Oct 29 22:42:33.270669 ip-10-0-135-80 kubenswrapper[2581]: I1029 22:42:33.270487 2581 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:42:33.270675 ip-10-0-135-80 systemd[1]: Stopping Kubernetes Kubelet... Oct 29 22:42:33.285519 ip-10-0-135-80 systemd[1]: kubelet.service: Deactivated successfully. Oct 29 22:42:33.285684 ip-10-0-135-80 systemd[1]: Stopped Kubernetes Kubelet. Oct 29 22:42:33.285848 ip-10-0-135-80 systemd[1]: kubelet.service: Consumed 3.097s CPU time. Oct 29 22:42:33.302905 ip-10-0-135-80 systemd[1]: Starting Kubernetes Kubelet... Oct 29 22:42:33.392840 ip-10-0-135-80 kubenswrapper[5057]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:42:33.392840 ip-10-0-135-80 kubenswrapper[5057]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 22:42:33.392840 ip-10-0-135-80 kubenswrapper[5057]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:42:33.392840 ip-10-0-135-80 kubenswrapper[5057]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:42:33.394959 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.394873 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:33.394959 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.394958 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:33.394959 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.394962 5057 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:33.394959 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.394964 5057 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:33.395109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.394968 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:33.395109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.394971 5057 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:33.395109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.394975 5057 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:33.395109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.394977 5057 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:33.395109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.394980 5057 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:33.395109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.394983 5057 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:33.395109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.394986 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:33.395109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.394988 5057 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:33.395109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.394991 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:33.395109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.394994 5057 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:33.395109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.394997 5057 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:33.395109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.394999 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:33.395109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395002 5057 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:33.395109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395004 5057 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:33.395109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395007 5057 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:33.395109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395010 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:33.395109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395012 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:33.395109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395015 5057 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:33.395109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395018 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:33.395109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395021 5057 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:33.395654 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395025 5057 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:33.395654 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395029 5057 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:33.395654 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395033 5057 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:33.395654 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395041 5057 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:33.395654 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395045 5057 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:33.395654 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395047 5057 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:33.395654 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395050 5057 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:33.395654 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395053 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:33.395654 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395056 5057 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:33.395654 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395058 5057 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:33.395654 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395061 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:33.395654 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395063 5057 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:33.395654 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395066 5057 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:33.395654 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395069 5057 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:33.395654 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395072 5057 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:33.395654 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395076 5057 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:33.395654 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395080 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:33.395654 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395083 5057 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:33.395654 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395086 5057 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:33.396136 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395089 5057 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:33.396136 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395092 5057 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:33.396136 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395097 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:33.396136 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395100 5057 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:33.396136 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395102 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:33.396136 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395105 5057 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:33.396136 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395108 5057 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:33.396136 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395110 5057 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:33.396136 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395113 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:33.396136 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395116 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:33.396136 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395120 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:33.396136 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395123 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:33.396136 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395126 5057 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:33.396136 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395128 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:33.396136 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395131 5057 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:33.396136 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395134 5057 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:33.396136 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395137 5057 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:33.396136 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395140 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:33.396136 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395143 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:33.396136 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395146 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:33.396661 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395148 5057 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:33.396661 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395151 5057 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:33.396661 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395153 5057 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:33.396661 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395156 5057 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:33.396661 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395158 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:33.396661 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395161 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:33.396661 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395164 5057 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:33.396661 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395166 5057 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:33.396661 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395169 5057 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:33.396661 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395172 5057 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:33.396661 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395174 5057 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:33.396661 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395177 5057 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:33.396661 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395179 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:33.396661 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395182 5057 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:33.396661 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395203 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:33.396661 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395206 5057 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:33.396661 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395209 5057 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:33.396661 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395212 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:33.396661 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395214 5057 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:33.396661 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395217 5057 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:33.397159 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395220 5057 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:33.397159 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395223 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:33.397159 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395226 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:33.397159 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395229 5057 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:33.397159 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395232 5057 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:33.397159 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395234 5057 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:33.397159 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395237 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:33.397159 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395240 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:33.397159 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395242 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:33.397159 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395245 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:33.397159 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395248 5057 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:33.397159 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395250 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:33.397159 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395253 5057 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:33.397159 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395255 5057 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:33.397159 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395258 5057 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:33.397159 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395261 5057 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:33.397159 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395264 5057 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:33.397159 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395267 5057 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:33.397159 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395269 5057 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:33.397667 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395272 5057 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:33.397667 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395274 5057 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:33.397667 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395277 5057 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:33.397667 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.395279 5057 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:33.397667 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396094 5057 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:33.397667 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396102 5057 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:33.397667 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396105 5057 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:33.397667 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396108 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:33.397667 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396111 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:33.397667 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396114 5057 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:33.397667 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396116 5057 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:33.397667 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396119 5057 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:33.397667 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396121 5057 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:33.397667 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396124 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:33.397667 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396127 5057 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:33.397667 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396130 5057 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:33.397667 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396133 5057 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:33.397667 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396135 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:33.397667 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396138 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:33.397667 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396141 5057 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:33.398167 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396144 5057 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:33.398167 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396147 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:33.398167 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396150 5057 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:33.398167 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396152 5057 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:33.398167 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396155 5057 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:33.398167 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396157 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:33.398167 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396160 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:33.398167 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396163 5057 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:33.398167 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396165 5057 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:33.398167 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396168 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:33.398167 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396171 5057 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:33.398167 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396173 5057 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:33.398167 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396175 5057 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:33.398167 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396178 5057 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:33.398167 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396181 5057 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:33.398167 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396198 5057 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:33.398167 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396201 5057 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:33.398167 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396205 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:33.398167 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396208 5057 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:33.398167 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396210 5057 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:33.398699 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396213 5057 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:33.398699 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396216 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:33.398699 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396222 5057 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:33.398699 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396225 5057 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:33.398699 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396228 5057 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:33.398699 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396231 5057 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:33.398699 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396233 5057 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:33.398699 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396236 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:33.398699 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396239 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:33.398699 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396242 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:33.398699 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396244 5057 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:33.398699 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396247 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:33.398699 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396249 5057 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:33.398699 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396252 5057 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:33.398699 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396254 5057 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:33.398699 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396257 5057 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:33.398699 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396260 5057 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:33.398699 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396264 5057 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:33.398699 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396268 5057 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:33.399181 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396271 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:33.399181 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396273 5057 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:33.399181 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396276 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:33.399181 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396279 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:33.399181 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396281 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:33.399181 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396284 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:33.399181 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396286 5057 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:33.399181 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396289 5057 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:33.399181 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396294 5057 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:33.399181 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396298 5057 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:33.399181 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396301 5057 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:33.399181 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396304 5057 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:33.399181 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396307 5057 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:33.399181 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396310 5057 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:33.399181 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396312 5057 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:33.399181 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396315 5057 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:33.399181 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396318 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:33.399181 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396321 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:33.399181 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396323 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:33.399694 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396326 5057 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:33.399694 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396329 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:33.399694 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396332 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:33.399694 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396334 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:33.399694 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396337 5057 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:33.399694 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396340 5057 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:33.399694 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396342 5057 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:33.399694 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396345 5057 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:33.399694 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396347 5057 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:33.399694 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396350 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:33.399694 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396352 5057 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:33.399694 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396355 5057 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:33.399694 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396357 5057 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:33.399694 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396360 5057 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:33.399694 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396362 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:33.399694 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396365 5057 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:33.399694 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396367 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:33.399694 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396370 5057 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:33.399694 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396372 5057 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:33.400938 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396375 5057 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:33.400938 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396377 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:33.400938 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396380 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:33.400938 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396383 5057 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:33.400938 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396385 5057 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:33.400938 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396388 5057 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:33.400938 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396390 5057 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:33.400938 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396393 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:33.400938 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396396 5057 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:33.400938 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396398 5057 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:33.400938 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396401 5057 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:33.400938 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396408 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:33.400938 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396412 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:33.400938 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396873 5057 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:33.400938 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396878 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:33.400938 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396881 5057 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:33.400938 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396883 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:33.400938 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396886 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:33.400938 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396889 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:33.400938 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396891 5057 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:33.401470 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396894 5057 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:33.401470 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396897 5057 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:33.401470 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396899 5057 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:33.401470 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396902 5057 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:33.401470 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396904 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:33.401470 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396907 5057 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:33.401470 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396910 5057 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:33.401470 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396912 5057 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:33.401470 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396916 5057 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:33.401470 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396921 5057 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:33.401470 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396924 5057 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:33.401470 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396927 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:33.401470 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396930 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:33.401470 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396934 5057 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:33.401470 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396937 5057 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:33.401470 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396940 5057 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:33.401470 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396943 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:33.401470 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396945 5057 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:33.401470 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396949 5057 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:33.401977 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396951 5057 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:33.401977 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396954 5057 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:33.401977 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396957 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:33.401977 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396959 5057 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:33.401977 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396963 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:33.401977 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396965 5057 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:33.401977 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396970 5057 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:33.401977 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396973 5057 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:33.401977 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396975 5057 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:33.401977 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396978 5057 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:33.401977 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396980 5057 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:33.401977 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396983 5057 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:33.401977 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396986 5057 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:33.401977 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396988 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:33.401977 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396991 5057 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:33.401977 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396993 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:33.401977 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396996 5057 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:33.401977 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.396998 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:33.401977 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397002 5057 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:33.401977 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397004 5057 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:33.402481 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397007 5057 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:33.402481 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397010 5057 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:33.402481 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397013 5057 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:33.402481 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397015 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:33.402481 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397018 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:33.402481 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397020 5057 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:33.402481 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397023 5057 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:33.402481 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397025 5057 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:33.402481 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397028 5057 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:33.402481 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397031 5057 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:33.402481 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397033 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:33.402481 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397036 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:33.402481 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397038 5057 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:33.402481 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397041 5057 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:33.402481 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397043 5057 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:33.402481 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397046 5057 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:33.402481 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397048 5057 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:33.402481 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397051 5057 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:33.402481 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397059 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:33.402996 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397062 5057 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:33.402996 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397065 5057 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:33.402996 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397068 5057 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:33.402996 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397070 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:33.402996 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397073 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:33.402996 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397075 5057 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:33.402996 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397078 5057 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:33.402996 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397081 5057 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:33.402996 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397083 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:33.402996 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397086 5057 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:33.402996 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397089 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:33.402996 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397092 5057 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:33.402996 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397096 5057 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:33.402996 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397100 5057 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:33.402996 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397103 5057 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:33.402996 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397105 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:33.402996 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397109 5057 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:33.402996 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397111 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:33.402996 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397114 5057 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:33.402996 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397116 5057 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:33.403518 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397119 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:33.403518 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397121 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:33.403518 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397124 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:33.403518 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397126 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:33.403518 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397129 5057 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:33.403518 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397131 5057 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:33.403518 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397134 5057 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:33.403518 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397137 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:33.403518 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397139 5057 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:33.403518 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397142 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:33.403518 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397144 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:33.403518 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397148 5057 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:33.403518 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397151 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:33.403518 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397155 5057 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:33.403518 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397157 5057 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:33.403518 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397160 5057 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:33.403518 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397162 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:33.403518 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397165 5057 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:33.403518 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397171 5057 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397173 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397176 5057 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397280 5057 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397287 5057 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397295 5057 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397302 5057 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397309 5057 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397314 5057 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397320 5057 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397324 5057 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397328 5057 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397331 5057 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397334 5057 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397338 5057 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397341 5057 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397344 5057 flags.go:64] FLAG: --cgroup-root="" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397347 5057 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397350 5057 flags.go:64] FLAG: --client-ca-file="" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397353 5057 flags.go:64] FLAG: --cloud-provider="external" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397356 5057 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397360 5057 flags.go:64] FLAG: --cluster-domain="" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397364 5057 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397367 5057 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 22:42:33.403978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397370 5057 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397375 5057 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397380 5057 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397383 5057 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397386 5057 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397389 5057 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397392 5057 flags.go:64] FLAG: --contention-profiling="false" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397395 5057 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397400 5057 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397403 5057 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397406 5057 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397410 5057 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397413 5057 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397416 5057 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397419 5057 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397422 5057 flags.go:64] FLAG: --enable-server="true" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397426 5057 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397430 5057 flags.go:64] FLAG: --event-burst="100" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397433 5057 flags.go:64] FLAG: --event-qps="50" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397436 5057 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397439 5057 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397442 5057 flags.go:64] FLAG: --eviction-hard="" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397447 5057 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397450 5057 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397453 5057 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 22:42:33.404587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397456 5057 flags.go:64] FLAG: --eviction-soft="" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397459 5057 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397463 5057 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397466 5057 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397469 5057 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397472 5057 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397474 5057 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397478 5057 flags.go:64] FLAG: --feature-gates="" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397483 5057 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397486 5057 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397489 5057 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397493 5057 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397496 5057 flags.go:64] FLAG: --healthz-port="10248" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397499 5057 flags.go:64] FLAG: --help="false" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397502 5057 flags.go:64] FLAG: --hostname-override="ip-10-0-135-80.ec2.internal" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397506 5057 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397510 5057 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397513 5057 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397516 5057 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397520 5057 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397523 5057 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397526 5057 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397528 5057 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 22:42:33.405232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397531 5057 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397534 5057 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397538 5057 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397540 5057 flags.go:64] FLAG: --kube-reserved="" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397544 5057 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397547 5057 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397550 5057 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397553 5057 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397555 5057 flags.go:64] FLAG: --lock-file="" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397558 5057 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397561 5057 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397564 5057 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397570 5057 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397572 5057 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397575 5057 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397578 5057 flags.go:64] FLAG: --logging-format="text" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397581 5057 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397586 5057 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397589 5057 flags.go:64] FLAG: --manifest-url="" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397592 5057 flags.go:64] FLAG: --manifest-url-header="" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397597 5057 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397600 5057 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397604 5057 flags.go:64] FLAG: --max-pods="110" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397607 5057 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397611 5057 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 22:42:33.405805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397614 5057 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397617 5057 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397620 5057 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397623 5057 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397626 5057 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397634 5057 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397637 5057 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397640 5057 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397643 5057 flags.go:64] FLAG: --pod-cidr="" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397646 5057 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397649 5057 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397652 5057 flags.go:64] FLAG: --pods-per-core="0" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397655 5057 flags.go:64] FLAG: --port="10250" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397658 5057 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397661 5057 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01971e8024657c89d" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397664 5057 flags.go:64] FLAG: --qos-reserved="" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397667 5057 flags.go:64] FLAG: --read-only-port="10255" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397670 5057 flags.go:64] FLAG: --register-node="true" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397673 5057 flags.go:64] FLAG: --register-with-taints="" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397677 5057 flags.go:64] FLAG: --registry-burst="10" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397680 5057 flags.go:64] FLAG: --registry-qps="5" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397683 5057 flags.go:64] FLAG: --reserved-cpus="" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397686 5057 flags.go:64] FLAG: --reserved-memory="" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397690 5057 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397694 5057 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397697 5057 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 22:42:33.406853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397700 5057 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397703 5057 flags.go:64] FLAG: --runonce="false" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397707 5057 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397711 5057 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397714 5057 flags.go:64] FLAG: --seccomp-default="false" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397717 5057 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397721 5057 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397724 5057 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397727 5057 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397730 5057 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397733 5057 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397736 5057 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397739 5057 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397742 5057 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397745 5057 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397748 5057 flags.go:64] FLAG: --system-cgroups="" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397751 5057 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397756 5057 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397759 5057 flags.go:64] FLAG: --tls-cert-file="" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397762 5057 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397766 5057 flags.go:64] FLAG: --tls-min-version="" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397769 5057 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397772 5057 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397775 5057 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397777 5057 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 22:42:33.407778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397780 5057 flags.go:64] FLAG: --v="2" Oct 29 22:42:33.408424 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397784 5057 flags.go:64] FLAG: --version="false" Oct 29 22:42:33.408424 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397789 5057 flags.go:64] FLAG: --vmodule="" Oct 29 22:42:33.408424 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397793 5057 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 22:42:33.408424 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.397796 5057 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 22:42:33.408424 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397924 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:33.408424 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397928 5057 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:33.408424 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397931 5057 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:33.408424 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397934 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:33.408424 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397936 5057 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:33.408424 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397940 5057 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:33.408424 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397942 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:33.408424 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397945 5057 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:33.408424 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397948 5057 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:33.408424 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397950 5057 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:33.408424 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397953 5057 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:33.408424 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397955 5057 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:33.408424 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397958 5057 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:33.408424 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397961 5057 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:33.408424 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397963 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:33.408424 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397967 5057 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:33.408984 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397971 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:33.408984 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397974 5057 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:33.408984 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397976 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:33.408984 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397979 5057 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:33.408984 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397982 5057 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:33.408984 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397984 5057 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:33.408984 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397987 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:33.408984 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397989 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:33.408984 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397992 5057 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:33.408984 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397994 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:33.408984 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397997 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:33.408984 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.397999 5057 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:33.408984 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398002 5057 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:33.408984 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398005 5057 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:33.408984 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398008 5057 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:33.408984 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398010 5057 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:33.408984 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398014 5057 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:33.408984 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398018 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:33.408984 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398021 5057 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:33.409572 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398024 5057 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:33.409572 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398026 5057 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:33.409572 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398029 5057 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:33.409572 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398032 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:33.409572 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398034 5057 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:33.409572 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398037 5057 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:33.409572 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398039 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:33.409572 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398042 5057 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:33.409572 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398044 5057 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:33.409572 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398047 5057 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:33.409572 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398050 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:33.409572 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398052 5057 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:33.409572 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398055 5057 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:33.409572 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398058 5057 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:33.409572 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398060 5057 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:33.409572 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398063 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:33.409572 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398065 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:33.409572 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398068 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:33.409572 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398070 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:33.409572 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398073 5057 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:33.410214 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398075 5057 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:33.410214 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398078 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:33.410214 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398080 5057 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:33.410214 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398083 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:33.410214 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398086 5057 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:33.410214 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398088 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:33.410214 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398091 5057 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:33.410214 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398093 5057 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:33.410214 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398096 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:33.410214 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398098 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:33.410214 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398101 5057 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:33.410214 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398103 5057 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:33.410214 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398106 5057 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:33.410214 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398108 5057 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:33.410214 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398110 5057 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:33.410214 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398113 5057 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:33.410214 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398116 5057 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:33.410214 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398118 5057 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:33.410214 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398121 5057 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:33.410214 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398123 5057 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:33.410885 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398126 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:33.410885 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398130 5057 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:33.410885 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398133 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:33.410885 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398135 5057 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:33.410885 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398139 5057 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:33.410885 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398142 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:33.410885 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398144 5057 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:33.410885 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398147 5057 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:33.410885 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398150 5057 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:33.410885 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398152 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:33.410885 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398155 5057 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:33.410885 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398158 5057 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:33.410885 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398160 5057 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:33.410885 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398163 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:33.410885 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398166 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:33.410885 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398168 5057 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:33.410885 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398171 5057 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:33.410885 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398174 5057 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:33.410885 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398176 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:33.411409 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398179 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:33.411409 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398182 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:33.411409 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398198 5057 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:33.411409 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398200 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:33.411409 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398203 5057 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:33.411409 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398206 5057 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:33.411409 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398208 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:33.411409 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398211 5057 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:33.411409 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398214 5057 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:33.411409 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398216 5057 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:33.411409 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398219 5057 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:33.411409 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398222 5057 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:33.411409 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.398248 5057 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:42:33.411409 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398282 5057 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:33.411409 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398286 5057 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:33.411409 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398291 5057 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:33.411833 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398293 5057 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:33.411833 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398298 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:33.411833 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398300 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:33.411833 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398303 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:33.411833 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398306 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:33.411833 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398308 5057 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:33.411833 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398311 5057 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:33.411833 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398314 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:33.411833 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398316 5057 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:33.411833 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398319 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:33.411833 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398322 5057 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:33.411833 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398325 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:33.411833 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398327 5057 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:33.411833 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398330 5057 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:33.411833 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398333 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:33.411833 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398335 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:33.411833 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398338 5057 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:33.411833 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398341 5057 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:33.411833 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398343 5057 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:33.411833 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398346 5057 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:33.412499 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398348 5057 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:33.412499 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398351 5057 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:33.412499 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398353 5057 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:33.412499 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398356 5057 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:33.412499 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398358 5057 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:33.412499 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398361 5057 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:33.412499 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398364 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:33.412499 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398367 5057 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:33.412499 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398369 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:33.412499 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398372 5057 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:33.412499 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398374 5057 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:33.412499 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398378 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:33.412499 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398381 5057 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:33.412499 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398385 5057 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:33.412499 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398387 5057 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:33.412499 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398390 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:33.412499 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398392 5057 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:33.412499 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398395 5057 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:33.412499 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398398 5057 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:33.412499 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398401 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:33.413072 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398403 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:33.413072 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398406 5057 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:33.413072 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398408 5057 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:33.413072 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398411 5057 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:33.413072 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398414 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:33.413072 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398416 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:33.413072 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398419 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:33.413072 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398422 5057 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:33.413072 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398424 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:33.413072 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398427 5057 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:33.413072 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398429 5057 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:33.413072 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398432 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:33.413072 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398434 5057 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:33.413072 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398437 5057 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:33.413072 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398439 5057 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:33.413072 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398442 5057 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:33.413072 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398444 5057 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:33.413072 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398448 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:33.413072 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398450 5057 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:33.413639 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398453 5057 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:33.413639 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398456 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:33.413639 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398458 5057 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:33.413639 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398461 5057 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:33.413639 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398465 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:33.413639 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398467 5057 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:33.413639 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398473 5057 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:33.413639 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398476 5057 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:33.413639 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398478 5057 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:33.413639 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398481 5057 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:33.413639 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398484 5057 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:33.413639 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398486 5057 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:33.413639 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398489 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:33.413639 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398492 5057 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:33.413639 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398496 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:33.413639 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398499 5057 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:33.413639 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398502 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:33.413639 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398513 5057 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:33.413639 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398517 5057 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:33.414106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398520 5057 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:33.414106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398523 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:33.414106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398525 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:33.414106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398528 5057 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:33.414106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398531 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:33.414106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398533 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:33.414106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398536 5057 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:33.414106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398539 5057 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:33.414106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398541 5057 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:33.414106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398544 5057 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:33.414106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398547 5057 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:33.414106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398551 5057 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:33.414106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398555 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:33.414106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398557 5057 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:33.414106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398560 5057 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:33.414106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398563 5057 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:33.414106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398566 5057 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:33.414106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398570 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:33.414106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398573 5057 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:33.414625 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398577 5057 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:33.414625 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398580 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:33.414625 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398582 5057 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:33.414625 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398585 5057 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:33.414625 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398588 5057 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:33.414625 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.398591 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:33.414625 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.402836 5057 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 22:42:33.414625 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.402851 5057 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 22:42:33.414625 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.402868 5057 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 22:42:33.414625 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.402873 5057 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 22:42:33.414625 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.402934 5057 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:33.414625 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.402939 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:33.414625 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.402942 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:33.414625 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.402945 5057 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:33.414625 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.402948 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:33.414625 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.402951 5057 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:33.414625 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.402953 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:33.414625 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.402956 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:33.414625 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.402959 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:33.414625 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.402962 5057 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:33.415156 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.402966 5057 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:33.415156 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.402970 5057 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:33.415156 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.402974 5057 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:33.415156 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.402979 5057 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:33.415156 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.402983 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:33.415156 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.402986 5057 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:33.415156 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.402990 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:33.415156 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.402992 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:33.415156 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403002 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:33.415156 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403005 5057 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:33.415156 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403010 5057 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:33.415156 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403014 5057 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:33.415156 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403018 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:33.415156 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403021 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:33.415156 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403025 5057 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:33.415156 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403028 5057 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:33.415156 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403031 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:33.415156 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403034 5057 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:33.415156 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403036 5057 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:33.415664 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403039 5057 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:33.415664 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403042 5057 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:33.415664 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403045 5057 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:33.415664 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403047 5057 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:33.415664 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403050 5057 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:33.415664 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403053 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:33.415664 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403055 5057 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:33.415664 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403058 5057 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:33.415664 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403060 5057 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:33.415664 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403063 5057 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:33.415664 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403066 5057 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:33.415664 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403069 5057 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:33.415664 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403071 5057 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:33.415664 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403074 5057 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:33.415664 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403077 5057 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:33.415664 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403080 5057 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:33.415664 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403082 5057 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:33.415664 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403085 5057 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:33.415664 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403088 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:33.415664 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403091 5057 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:33.416145 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403093 5057 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:33.416145 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403096 5057 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:33.416145 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403104 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:33.416145 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403108 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:33.416145 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403110 5057 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:33.416145 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403114 5057 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:33.416145 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403116 5057 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:33.416145 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403119 5057 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:33.416145 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403122 5057 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:33.416145 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403124 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:33.416145 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403127 5057 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:33.416145 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403129 5057 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:33.416145 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403132 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:33.416145 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403135 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:33.416145 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403137 5057 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:33.416145 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403140 5057 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:33.416145 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403143 5057 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:33.416145 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403145 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:33.416145 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403148 5057 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:33.416145 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403150 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:33.416649 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403153 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:33.416649 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403155 5057 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:33.416649 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403159 5057 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:33.416649 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403163 5057 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:33.416649 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403166 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:33.416649 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403169 5057 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:33.416649 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403171 5057 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:33.416649 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403174 5057 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:33.416649 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403176 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:33.416649 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403179 5057 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:33.416649 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403182 5057 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:33.416649 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403196 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:33.416649 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403200 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:33.416649 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403202 5057 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:33.416649 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403205 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:33.416649 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403214 5057 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:33.416649 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403217 5057 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:33.416649 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403220 5057 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:33.416649 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403223 5057 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:33.417106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403226 5057 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:33.417106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403229 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:33.417106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403231 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:33.417106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403234 5057 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:33.417106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403237 5057 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:33.417106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403240 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:33.417106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403242 5057 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:33.417106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403245 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:33.417106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403247 5057 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:33.417106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403250 5057 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:33.417106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403253 5057 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:33.417106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403255 5057 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:33.417106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403258 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:33.417106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403260 5057 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:33.417106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403263 5057 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:33.417106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403266 5057 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:33.417106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403269 5057 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:33.417106 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403271 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:33.417551 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.403299 5057 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:42:33.417551 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403503 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:33.417551 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403508 5057 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:33.417551 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403511 5057 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:33.417551 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403514 5057 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:33.417551 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403517 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:33.417551 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403520 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:33.417551 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403523 5057 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:33.417551 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403526 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:33.417551 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403528 5057 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:33.417551 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403531 5057 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:33.417551 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403540 5057 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:33.417551 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403543 5057 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:33.417551 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403545 5057 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:33.417551 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403548 5057 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:33.417551 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403551 5057 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:33.417943 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403553 5057 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:33.417943 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403556 5057 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:33.417943 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403559 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:33.417943 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403561 5057 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:33.417943 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403563 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:33.417943 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403566 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:33.417943 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403569 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:33.417943 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403571 5057 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:33.417943 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403574 5057 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:33.417943 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403577 5057 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:33.417943 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403580 5057 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:33.417943 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403583 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:33.417943 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403585 5057 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:33.417943 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403588 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:33.417943 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403590 5057 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:33.417943 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403593 5057 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:33.417943 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403596 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:33.417943 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403598 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:33.417943 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403601 5057 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:33.418428 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403604 5057 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:33.418428 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403607 5057 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:33.418428 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403609 5057 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:33.418428 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403613 5057 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:33.418428 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403616 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:33.418428 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403619 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:33.418428 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403624 5057 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:33.418428 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403627 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:33.418428 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403630 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:33.418428 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403642 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:33.418428 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403645 5057 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:33.418428 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403647 5057 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:33.418428 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403650 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:33.418428 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403652 5057 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:33.418428 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403656 5057 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:33.418428 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403658 5057 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:33.418428 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403661 5057 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:33.418428 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403664 5057 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:33.418428 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403666 5057 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:33.418428 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403669 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:33.419001 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403672 5057 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:33.419001 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403678 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:33.419001 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403681 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:33.419001 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403683 5057 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:33.419001 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403686 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:33.419001 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403689 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:33.419001 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403691 5057 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:33.419001 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403694 5057 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:33.419001 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403697 5057 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:33.419001 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403699 5057 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:33.419001 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403702 5057 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:33.419001 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403705 5057 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:33.419001 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403707 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:33.419001 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403710 5057 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:33.419001 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403712 5057 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:33.419001 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403715 5057 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:33.419001 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403719 5057 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:33.419001 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403723 5057 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:33.419001 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403729 5057 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:33.419563 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403732 5057 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:33.419563 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403735 5057 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:33.419563 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403738 5057 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:33.419563 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403748 5057 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:33.419563 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403751 5057 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:33.419563 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403759 5057 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:33.419563 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403762 5057 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:33.419563 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403765 5057 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:33.419563 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403768 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:33.419563 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403770 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:33.419563 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403773 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:33.419563 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403776 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:33.419563 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403778 5057 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:33.419563 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403781 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:33.419563 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403785 5057 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:33.419563 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403788 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:33.419563 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403790 5057 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:33.419563 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403793 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:33.419563 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403796 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:33.420071 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403798 5057 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:33.420071 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403801 5057 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:33.420071 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403804 5057 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:33.420071 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403807 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:33.420071 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403809 5057 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:33.420071 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403812 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:33.420071 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403814 5057 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:33.420071 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403817 5057 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:33.420071 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403820 5057 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:33.420071 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403823 5057 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:33.420071 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403825 5057 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:33.420071 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403828 5057 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:33.420071 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403832 5057 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:33.420071 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403835 5057 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:33.420071 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.403860 5057 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:42:33.420071 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403901 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:33.420466 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403905 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:33.420466 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403915 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:33.420466 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403918 5057 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:33.420466 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403921 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:33.420466 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403923 5057 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:33.420466 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403926 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:33.420466 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403929 5057 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:33.420466 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403931 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:33.420466 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403934 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:33.420466 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403936 5057 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:33.420466 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403939 5057 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:33.420466 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403942 5057 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:33.420466 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403946 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:33.420466 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403949 5057 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:33.420466 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403951 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:33.420466 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403954 5057 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:33.420466 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403956 5057 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:33.420466 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403959 5057 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:33.420466 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403962 5057 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:33.420926 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403964 5057 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:33.420926 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403967 5057 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:33.420926 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403969 5057 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:33.420926 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403972 5057 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:33.420926 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403975 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:33.420926 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403977 5057 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:33.420926 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403980 5057 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:33.420926 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403983 5057 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:33.420926 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403986 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:33.420926 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403990 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:33.420926 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403992 5057 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:33.420926 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403995 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:33.420926 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.403998 5057 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:33.420926 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404000 5057 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:33.420926 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404003 5057 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:33.420926 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404011 5057 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:33.420926 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404014 5057 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:33.420926 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404016 5057 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:33.420926 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404019 5057 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:33.420926 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404021 5057 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:33.421462 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404024 5057 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:33.421462 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404027 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:33.421462 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404029 5057 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:33.421462 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404032 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:33.421462 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404035 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:33.421462 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404038 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:33.421462 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404041 5057 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:33.421462 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404044 5057 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:33.421462 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404046 5057 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:33.421462 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404049 5057 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:33.421462 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404052 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:33.421462 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404054 5057 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:33.421462 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404057 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:33.421462 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404059 5057 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:33.421462 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404062 5057 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:33.421462 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404065 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:33.421462 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404067 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:33.421462 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404070 5057 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:33.421462 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404073 5057 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:33.421918 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404076 5057 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:33.421918 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404078 5057 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:33.421918 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404084 5057 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:33.421918 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404087 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:33.421918 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404099 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:33.421918 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404102 5057 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:33.421918 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404104 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:33.421918 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404107 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:33.421918 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404110 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:33.421918 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404118 5057 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:33.421918 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404121 5057 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:33.421918 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404124 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:33.421918 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404127 5057 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:33.421918 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404129 5057 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:33.421918 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404132 5057 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:33.421918 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404134 5057 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:33.421918 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404138 5057 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:33.421918 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404141 5057 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:33.421918 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404145 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:33.421918 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404148 5057 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:33.422539 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404150 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:33.422539 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404153 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:33.422539 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404155 5057 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:33.422539 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404158 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:33.422539 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404161 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:33.422539 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404164 5057 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:33.422539 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404166 5057 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:33.422539 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404169 5057 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:33.422539 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404172 5057 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:33.422539 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404174 5057 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:33.422539 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404177 5057 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:33.422539 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404180 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:33.422539 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404182 5057 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:33.422539 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404199 5057 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:33.422539 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404204 5057 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:33.422539 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404208 5057 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:33.422539 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404212 5057 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:33.422539 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404215 5057 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:33.422539 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404219 5057 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:33.423040 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404222 5057 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:33.423040 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404225 5057 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:33.423040 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404229 5057 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:33.423040 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404240 5057 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:33.423040 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404243 5057 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:33.423040 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404246 5057 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:33.423040 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404250 5057 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:33.423040 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:33.404253 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:33.423040 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.404305 5057 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 22:42:33.423040 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.405319 5057 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 29 22:42:33.423040 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.405387 5057 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 29 22:42:33.423040 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.405582 5057 server.go:1014] "Starting client certificate rotation" Oct 29 22:42:33.423040 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.405770 5057 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:42:33.423040 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.406066 5057 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:36:53 +0000 UTC" deadline="2028-05-13 08:27:04.366580055 +0000 UTC" Oct 29 22:42:33.423040 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.406697 5057 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13473h44m30.959891427s" Oct 29 22:42:33.423040 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.407102 5057 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:42:33.423455 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.408648 5057 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:42:33.423455 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.410171 5057 log.go:25] "Validated CRI v1 runtime API" Oct 29 22:42:33.423455 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.412005 5057 log.go:25] "Validated CRI v1 image API" Oct 29 22:42:33.423455 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.412815 5057 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 22:42:33.423455 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.414877 5057 fs.go:135] Filesystem UUIDs: map[1ecb3b4d-13ea-4249-8958-3396da766a7a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ac1434ae-05a1-4400-8478-7443405f9dc4:/dev/nvme0n1p4] Oct 29 22:42:33.423646 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.414902 5057 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/028b9bc723b018c49696c401214b8ec85c9b1f9d54c7f29a10ddfb3600d43b32/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/028b9bc723b018c49696c401214b8ec85c9b1f9d54c7f29a10ddfb3600d43b32/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/220631aa8bf08160ea2ae099ede645cd10b43b7f584b3f3f159920391e1cc22d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/220631aa8bf08160ea2ae099ede645cd10b43b7f584b3f3f159920391e1cc22d/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25d1297cee065faf2844f297078cbe34338e029f0120e4f6fbadc072a2e4f1e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25d1297cee065faf2844f297078cbe34338e029f0120e4f6fbadc072a2e4f1e9/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ce7e52911f12103f3bf53cd734767cbff8b8fbdede8860e494cbcfc6f5813b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ce7e52911f12103f3bf53cd734767cbff8b8fbdede8860e494cbcfc6f5813b7/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e0e85234849cda82191e957089de80d763789a100521253717a21405e99c869/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e0e85234849cda82191e957089de80d763789a100521253717a21405e99c869/userdata/shm major:0 minor:319 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5148e4d802089db3f519fa75c16f6d11b1e8d116ca1337134eb4edf466cbacb6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5148e4d802089db3f519fa75c16f6d11b1e8d116ca1337134eb4edf466cbacb6/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/710f6482706cf21d13a0be40293e5c922dc9549d6d94c8e010c661f794105f8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/710f6482706cf21d13a0be40293e5c922dc9549d6d94c8e010c661f794105f8c/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72c20a30bd389edb03b22011f445c4d40c2fa5a93e32a280e4278313668c7aca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72c20a30bd389edb03b22011f445c4d40c2fa5a93e32a280e4278313668c7aca/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74d538b607b308700c2ddf7b631b7a4f89045fef1002029b6d323b160552ac49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74d538b607b308700c2ddf7b631b7a4f89045fef1002029b6d323b160552ac49/userdata/shm major:0 minor:347 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82c4b5ea11fe463ac3c5b009a989c3434ee67cf2d2db2628c885902825057b10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82c4b5ea11fe463ac3c5b009a989c3434ee67cf2d2db2628c885902825057b10/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b4ef5a62c85870b52d54ad3ffd257dee8b1784d7fd5cbbf259f4e8855eb7afb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b4ef5a62c85870b52d54ad3ffd257dee8b1784d7fd5cbbf259f4e8855eb7afb/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/901f5e90bef358345098700155850e5133969b5fd52e69a23d1452675c671c23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/901f5e90bef358345098700155850e5133969b5fd52e69a23d1452675c671c23/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9870c7f5e182878c16f0d6c428a1c62d407777effb29d920c0ffe042b0ed2873/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9870c7f5e182878c16f0d6c428a1c62d407777effb29d920c0ffe042b0ed2873/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99aaddab36725a8f6ffbbf5d65006c25fbd7bcca3de186b40c5774dbd8a614cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99aaddab36725a8f6ffbbf5d65006c25fbd7bcca3de186b40c5774dbd8a614cb/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f59a804858118aa038ff14cb50db8dcd52f6a3704bb85fbcf18447550d056fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f59a804858118aa038ff14cb50db8dcd52f6a3704bb85fbcf18447550d056fe/userdata/shm major:0 minor:323 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a289effe7cef40dca73995ac30e516b30174c991b7833f92e253ea116322a6d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a289effe7cef40dca73995ac30e516b30174c991b7833f92e253ea116322a6d6/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a37a3da87cc440a67f1cefaefb01ac48e701dfc430b6bce0687dd5782f6833d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a37a3da87cc440a67f1cefaefb01ac48e701dfc430b6bce0687dd5782f6833d7/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac9dd7d8666ac55fac7d0ebe766e087035079f15b2f3dcffcfb734b1f0597a10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac9dd7d8666ac55fac7d0ebe766e087035079f15b2f3dcffcfb734b1f0597a10/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0e0c2bbab3f446ae654fa789a3fdca7a20acb188bbae830f145505f196b75fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0e0c2bbab3f446ae654fa789a3fdca7a20acb188bbae830f145505f196b75fc/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b12aec2fedb8f3264afa0c6e4e36f3cc62676554e9adc0e336775257632dcf72/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b12aec2fedb8f3264afa0c6e4e36f3cc62676554e9adc0e336775257632dcf72/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cea9c5c68a1c41e3617d64636095622a1097edea881f16f4fc74dadeea8e15e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cea9c5c68a1c41e3617d64636095622a1097edea881f16f4fc74dadeea8e15e4/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e28f2f94763c91944b4b5e6fcdda5c2aa18ee62de01c0f5b3afbe0d2c67e887f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e28f2f94763c91944b4b5e6fcdda5c2aa18ee62de01c0f5b3afbe0d2c67e887f/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3629e1ac73011582ae081f05d21007b7c932456f98e64abbab89ce4fc161673/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3629e1ac73011582ae081f05d21007b7c932456f98e64abbab89ce4fc161673/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed05e90ad4316ad2bd053ae3fe901f90d0c3fe8f02ae86066c4e4e9f2729f8a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed05e90ad4316ad2bd053ae3fe901f90d0c3fe8f02ae86066c4e4e9f2729f8a6/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f94f8986e84080f4850d25025683022e39863ad901a59bfeb9eb02e89d5837da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f94f8986e84080f4850d25025683022e39863ad901a59bfeb9eb02e89d5837da/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd844cb3511c0f05c9f8c26c1c22d0494132ad245c74b1242938d0befbf75c70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd844cb3511c0f05c9f8c26c1c22d0494132ad245c74b1242938d0befbf75c70/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/093ceb90-294f-4133-83a6-32548405d530/volumes/kubernetes.io~projected/kube-api-access-5cffk:{mountpoint:/var/lib/kubelet/pods/093ceb90-294f-4133-83a6-32548405d530/volumes/kubernetes.io~projected/kube-api-access-5cffk major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/093ceb90-294f-4133-83a6-32548405d530/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/093ceb90-294f-4133-83a6-32548405d530/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/266bff54-6027-4822-91ce-be5cbd316ddc/volumes/kubernetes.io~projected/kube-api-access-82r6j:{mountpoint:/var/lib/kubelet/pods/266bff54-6027-4822-91ce-be5cbd316ddc/volumes/kubernetes.io~projected/kube-api-access-82r6j major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/286133c3-abe5-4954-83a1-8665c1785b92/volumes/kubernetes.io~projected/kube-api-access-qvrrh:{mountpoint:/var/lib/kubelet/pods/286133c3-abe5-4954-83a1-8665c1785b92/volumes/kubernetes.io~projected/kube-api-access-qvrrh major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a/volumes/kubernetes.io~projected/kube-api-access-85sqm:{mountpoint:/var/lib/kubelet/pods/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a/volumes/kubernetes.io~projected/kube-api-access-85sqm major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41ea529e-ab5d-419c-87bf-fe0bd1a6460f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/41ea529e-ab5d-419c-87bf-fe0bd1a6460f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4525bd80-124f-4894-9fc0-b8235a0fe689/volumes/kubernetes.io~projected/kube-api-access-gcb88:{mountpoint:/var/lib/kubelet/pods/4525bd80-124f-4894-9fc0-b8235a0fe689/volumes/kubernetes.io~projected/kube-api-access-gcb88 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4590699c-6eb3-4546-835f-924180048d8a/volumes/kubernetes.io~projected/kube-api-access-jdmbj:{mountpoint:/var/lib/kubelet/pods/4590699c-6eb3-4546-835f-924180048d8a/volumes/kubernetes.io~projected/kube-api-access-jdmbj major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/463bc340-b1dc-41e6-9203-fcd2672f8a17/volumes/kubernetes.io~projected/kube-api-access-jk8rz:{mountpoint:/var/lib/kubelet/pods/463bc340-b1dc-41e6-9203-fcd2672f8a17/volumes/kubernetes.io~projected/kube-api-access-jk8rz major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68daf26a-81a0-4bd5-a188-9b39a1f460a7/volumes/kubernetes.io~projected/kube-api-access-rz8wx:{mountpoint:/var/lib/kubelet/pods/68daf26a-81a0-4bd5-a188-9b39a1f460a7/volumes/kubernetes.io~projected/kube-api-access-rz8wx major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a4e4c73-0516-464c-9b6e-1c4adf93c03b/volumes/kubernetes.io~projected/kube-api-access-dcjhj:{mountpoint:/var/lib/kubelet/pods/6a4e4c73-0516-464c-9b6e-1c4adf93c03b/volumes/kubernetes.io~projected/kube-api-access-dcjhj major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71be8984-962e-4357-a596-b9d3931e3a75/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/71be8984-962e-4357-a596-b9d3931e3a75/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71be8984-962e-4357-a596-b9d3931e3a75/volumes/kubernetes.io~projected/kube-api-access-rhmrs:{mountpoint:/var/lib/kubelet/pods/71be8984-962e-4357-a596-b9d3931e3a75/volumes/kubernetes.io~projected/kube-api-access-rhmrs major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71be8984-962e-4357-a596-b9d3931e3a75/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/71be8984-962e-4357-a596-b9d3931e3a75/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73b41114-4ff0-4357-98b5-a871335ee3de/volumes/kubernetes.io~projected/kube-api-access-vfslj:{mountpoint:/var/lib/kubelet/pods/73b41114-4ff0-4357-98b5-a871335ee3de/volumes/kubernetes.io~projected/kube-api-access-vfslj major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73b41114-4ff0-4357-98b5-a871335ee3de/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/73b41114-4ff0-4357-98b5-a871335ee3de/volumes/kubernetes.io~secret/signing-key major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85325a00-d2ed-4c7d-b287-0fd5452f7e69/volumes/kubernetes.io~projected/kube-api-access-vrls7:{mountpoint:/var/lib/kubelet/pods/85325a00-d2ed-4c7d-b287-0fd5452f7e69/volumes/kubernetes.io~projected/kube-api-access-vrls7 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85ab246f-f679-40c5-a127-c0fd5a4a621e/volumes/kubernetes.io~projected/kube-api-access-g9l7w:{mountpoint:/var/lib/kubelet/pods/85ab246f-f679-40c5-a127-c0fd5a4a621e/volumes/kubernetes.io~projected/kube-api-access-g9l7w major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/877326a8-91a3-499f-8b37-4f80dfb00352/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/877326a8-91a3-499f-8b37-4f80dfb00352/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/877326a8-91a3-499f-8b37-4f80dfb00352/volumes/kubernetes.io~projected/kube-api-access-hcfvz:{mountpoint:/var/lib/kubelet/pods/877326a8-91a3-499f-8b37-4f80dfb00352/volumes/kubernetes.io~projected/kube-api-access-hcfvz major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/877326a8-91a3-499f-8b37-4f80dfb00352/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/877326a8-91a3-499f-8b37-4f80dfb00352/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/877326a8-91a3-499f-8b37-4f80dfb00352/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/877326a8-91a3-499f-8b37-4f80dfb00352/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/884370ff-c54f-4d6b-af78-5c0813bb48eb/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/884370ff-c54f-4d6b-af78-5c0813bb48eb/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/884370ff-c54f-4d6b-af78-5c0813bb48eb/volumes/kubernetes.io~projected/kube-api-access-t5l7q:{mountpoint:/var/lib/kubelet/pods/884370ff-c54f-4d6b-af78-5c0813bb48eb/volumes/kubernetes.io~projected/kube-api-access-t5l7q major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/884370ff-c54f-4d6b-af78-5c0813bb48eb/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/884370ff-c54f-4d6b-af78-5c0813bb48eb/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/884370ff-c54f-4d6b-af78-5c0813bb48eb/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/884370ff-c54f-4d6b-af78-5c0813bb48eb/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656/volumes/kubernetes.io~projected/kube-api-access-7hp7m:{mountpoint:/var/lib/kubelet/pods/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656/volumes/kubernetes.io~projected/kube-api-access-7hp7m major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f6c01b9-e044-413c-b533-09033d04f29e/volumes/kubernetes.io~projected/kube-api-access-m76zp:{mountpoint:/var/lib/kubelet/pods/8f6c01b9-e044-413c-b533-09033d04f29e/volumes/kubernetes.io~projected/kube-api-access-m76zp major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9483bf09-4676-47db-a279-84aabe9d17ed/volumes/kubernetes.io~projected/kube-api-access-w5d85:{mountpoint:/var/lib/kubelet/pods/9483bf09-4676-47db-a279-84aabe9d17ed/volumes/kubernetes.io~projected/kube-api-access-w5d85 major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96726184-9be2-4685-93d1-415bbb8b551c/volumes/kubernetes.io~projected/kube-api-access-cg6m4:{mountpoint:/var/lib/kubelet/pods/96726184-9be2-4685-93d1-415bbb8b551c/volumes/kubernetes.io~projected/kube-api-access-cg6m4 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97fa3566-0c3f-4768-a437-3a89816303e0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/97fa3566-0c3f-4768-a437-3a89816303e0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97fa3566-0c3f-4768-a437-3a89816303e0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/97fa3566-0c3f-4768-a437-3a89816303e0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97fa3566-0c3f-4768-a437-3a89816303e0/volumes/kubernetes.io~projected/kube-api-access-plrh7:{mountpoint:/var/lib/kubelet/pods/97fa3566-0c3f-4768-a437-3a89816303e0/volumes/kubernetes.io~projected/kube-api-access-plrh7 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99577b75-29cd-430b-b5ac-2af3828586d0/volumes/kubernetes.io~projected/kube-api-access-h744b:{mountpoint:/var/lib/kubelet/pods/99577b75-29cd-430b-b5ac-2af3828586d0/volumes/kubernetes.io~projected/kube-api-access-h744b major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99577b75-29cd-430b-b5ac-2af3828586d0/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/99577b75-29cd-430b-b5ac-2af3828586d0/volumes/kubernetes.io~secret/default-certificate major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99577b75-29cd-430b-b5ac-2af3828586d0/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/99577b75-29cd-430b-b5ac-2af3828586d0/volumes/kubernetes.io~secret/stats-auth major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e91a303-9b80-47f5-a502-fbb6365eaa54/volumes/kubernetes.io~projected/kube-api-access-zvm7r:{mountpoint:/var/lib/kubelet/pods/9e91a303-9b80-47f5-a502-fbb6365eaa54/volumes/kubernetes.io~projected/kube-api-access-zvm7r major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e91a303-9b80-47f5-a502-fbb6365eaa54/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9e91a303-9b80-47f5-a502-fbb6365eaa54/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a109715c-c29a-48d1-a573-3e7800f48259/volumes/kubernetes.io~projected/kube-api-access-zqxqz:{mountpoint:/var/lib/kubelet/pods/a109715c-c29a-48d1-a573-3e7800f48259/volumes/kubernetes.io~projected/kube-api-access-zqxqz major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a109715c-c29a-48d1-a573-3e7800f48259/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a109715c-c29a-48d1-a573-3e7800f48259/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1a232f4-f67c-404f-a1f7-af0c3cc30180/volumes/kubernetes.io~projected/kube-api-access-kgc6n:{mountpoint:/var/lib/kubelet/pods/a1a232f4-f67c-404f-a1f7-af0c3cc30180/volumes/kubernetes.io~projected/kube-api-access-kgc6n major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8/volumes/kubernetes.io~projected/kube-api-access-f6nxm:{mountpoint:/var/lib/kubelet/pods/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8/volumes/kubernetes.io~projected/kube-api-access-f6nxm major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad00084a-5a9f-489b-9122-2316f617f3f1/volumes/kubernetes.io~projected/kube-api-access-2kpn4:{mountpoint:/var/lib/kubelet/pods/ad00084a-5a9f-489b-9122-2316f617f3f1/volumes/kubernetes.io~projected/kube-api-access-2kpn4 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af926a18-1147-4786-8bd9-8a6727abb6fa/volumes/kubernetes.io~projected/kube-api-access-2z9zh:{mountpoint:/var/lib/kubelet/pods/af926a18-1147-4786-8bd9-8a6727abb6fa/volumes/kubernetes.io~projected/kube-api-access-2z9zh major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c29d1f02-5697-4d94-8a21-0a39e8f473ed/volumes/kubernetes.io~projected/kube-api-access-cx7m8:{mountpoint:/var/lib/kubelet/pods/c29d1f02-5697-4d94-8a21-0a39e8f473ed/volumes/kubernetes.io~projected/kube-api-access-cx7m8 major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce5dd070-fce7-44da-aaeb-017109fc98c9/volumes/kubernetes.io~projected/kube-api-access-h4z7h:{mountpoint:/var/lib/kubelet/pods/ce5dd070-fce7-44da-aaeb-017109fc98c9/volumes/kubernetes.io~projected/kube-api-access-h4z7h major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9/volumes/kubernetes.io~projected/kube-api-access-jrhns:{mountpoint:/var/lib/kubelet/pods/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9/volumes/kubernetes.io~projected/kube-api-access-jrhns major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6f91740-e20b-439e-be7d-4b444f4835fa/volumes/kubernetes.io~projected/kube-api-access-rx2rx:{mountpoint:/var/lib/kubelet/pods/d6f91740-e20b-439e-be7d-4b444f4835fa/volumes/kubernetes.io~projected/kube-api-access-rx2rx major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6f91740-e20b-439e-be7d-4b444f4835fa/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/d6f91740-e20b-439e-be7d-4b444f4835fa/volumes/kubernetes.io~secret/cert major:0 minor:205 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9c8207a1b6cae1a342b9cec14150a00eef08d268ff9e32e852dd93888d594089/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c6e4df968f7ea471b9d89fe9d30e8f35d5c68555da4b5e5c76924d1b7528b28b/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/669ea758d3a49bf18b8ba78c95d0e789b1ce36136c3e9c6189d1853eedd52989/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/ebbdfb3994c197daf1089f7a63f3fec4a5e3160a0d97f5f6f7730e89c2e24fcf/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/88f73c7937842154f2caa750cfef3d29bf7b62fa9cf57b128bd4fc2e5f8b2572/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/a1d9f0bfd37d808dbb14fd193f22f4250ff994c2e1e22fb7b9dee68ac2fabca4/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/9ef9a282f21e8579d6690a8cb2fad3203d559df5134cbb5e3777df30f625e821/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/21d48aee4d7fcb8c61e6950352f9e07553a4ada0e6340b74ad0e66ecfb3504cd/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/ea4f7ec67d035f53e1064bddfc1a97e7a250cb9b414abeb29545948681d80b18/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/e46792063137c34fc05c4d73089cc3ade7358c95a0065c4d46d042b9ac1888d6/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/228deb880cba024ce0c9e9f0b458864beb387f6dc2ef0a4721eb38fe7f7f726b/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/0be18182c98c448c0c748e401feca6ce4d7037d09ff8174c0ef1b3cc24f2903b/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/2ef9fce457a1e85498b0fa9d9bc74fb648129a6f09b786c53406cd9936787306/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/8d3376fac2e23c225b886cf5d3d3754d34dc94ed708ff7067bac87d925299e12/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/cfa06ed561ed302c92733cbc74ab17efaffaa4f0f749cd5acafd17c70ee948d3/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/7f9679f55bee6bab0987675727ab741e565d7eb0bc27b9bc387a865c7c0b0901/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/0b550c4c4f118042e94b22cb779c7383d83db993686b80d3b2e200651df1e68a/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/2d87be8cb5770fe1e881797d947fb9ce029e6cae99738ef6ba3f9f8ec80b36bc/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0db7967d7ec2183f0e4745e3a535a825197755bb386e6030d79aa17d803af777/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/6ba266b27fd829a035e4fe1c9ed6ea675ac129362bcae53d717998a447684c30/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/0ff11a4038c2a5aa30c3a987325a8818204ad8325d7d53ab983ee173779395b2/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/abe88dd7604c60ae19a8d1c695acf209be7bf670536e26f1df87a74a7de9983d/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/86ffc82602a623bf06001a251843f80516aada1b0cdf355a7a38293103c29146/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/952a204f734b563e8534dc279be1494a8d7756033418d800bbf935ec78c1eaa5/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/08672765e232f3fd91a8fba4c69cf7db7b22dd81dfb980634db97140360b4717/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/e52f43b91b7f0f770219a01592b5f5223347937445a7dfc18dc345fd9e0e419b/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/2217b1dd89f015ed47c5d8cbaca10ea3375757c2018ad710805b14face4b8491/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/f056357e4100174ba94d5c3bed5419616a0462353ac471a6ce1e952ce2a0f955/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/bdd82ae2d76d23c36754c392afc0461759f789896fd094b3b72c2d27bb500656/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/e5fe892ba620bebc6b769bb74ed3f9f497a46b97eea1f66ae292469eb723d26f/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/374451b7469518a7d5fcf61746fd11aadd4da7221011c7e0bd99400f85522790/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/f0693b40201e6532b9dbcf7d2f3baccab091ceff53615f7efa560f1b27dc063f/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/8e67d926a180897da86b09905a4188e4fb621efee7878d58ae88399a3c8c8d5c/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/53bf942bd23159391a4f04f4b1b28670ba3c37ee3a582a9c6dd9bf7f2a6ec8c4/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/449752d0ca96ece8a0c5e3d065d348269ea042f62145cdcc439c5ac2560f24c0/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/da68c7515ab26692eb20efd15827e8a7de222d7465da2d98c0b399c87b3e2523/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/857eec4b87d87cb2bf9a12ea7d553ce9490c76f9293f4e58f379fcfca66e1b8f/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/24975619934aca1607da984dcc45b28a712c0b538c682ee99189beb58161d200/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/2371de3b3ce668d292251a997c5d76ad26bb52933bbdc23b8bbb6ef7c513d7f7/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/89e3a44a2c7006b996cad9a517abddc6e7c019d79ac82ed566f92d5b5987c2f3/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/24e7316e44d1662b380ee0b0ec97aab8347c1fb6271624ef5df5461615ed687a/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/57083b8ba075adc5071e187897d77590bde92980d5948b6eb623436499e5a661/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/99ba0b6f2efc459d8734cbdb6896936ebfa1969f85c934fda86f9b854b69e750/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/8e659a5293248ccbec62037d806749ccd793e59f7e21542ec1987e98f4a9400d/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/dd484e78de7ebd1c5fa6b2c434e10363bf658153edfc1664ca99079b7b9f411e/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/47c0bef1d9b71c52764cb87fc00b01f71c1216219f0a8c16b183e9b6b5cb07ce/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/2a625ff610641053e817325da5c5585e9740b7bcf0c1beff50f6295cd8b5337c/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/f52c689cbd6ea2c06e6649a2469efbf95b518d94ed113ffb61f4c1da28a00058/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/3cf6662da85fed0b9716e973abc2db3b910e218a8e801c158886ddc6f6d3f9dc/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/be847423f628a64145bfb56f925ca771f4ff57618165ad383a1fc066f4f0be9b/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/18c912cfdcd3b1726dc42cc7412109af591bfa8ce967646b7b269a9b39b34a6a/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/d68e0aa57a0174525d2d6913bcd586f18d9d20a385b72287d5b8114ff4aed9ab/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/00a987d88657edd963ddafdc0c9118316ce456d83177d959970fc71fcae22c1f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/cc116c6eb1cdcb5af6d4d95a9b75b000c8782b566359e9c20c0762730345ae71/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/432452c68f46b6b29aa79efae57b0d0e32941e5724537a605aacd87a8f53980f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/29badc564348571f69bb2873a99ae89be7c5f0e881460f65ffaeeb6601a4757e/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/34f5aa1249883caa0547be1cd3989a39c75cf97dcd1d5087ee22370691af637a/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/a736b567d4752915cfcb7ce7b1b03fa30410aa1b6a4af7bf57d7e756086af17f/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/437ad9cb5bb6d7fd0c4505a3e347b54502b78275dc9c42616599fbb0bdca3acf/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/7caf5a897628049f54571d19bf88a77b94311ba341c94c7232fdf684a169839b/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/5d8e29ae3aceed7c1f85d9f72073d099f465c34555becdb19cbd4bd3f0a0f137/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/a7fbc5d31c5d26344b998e8ce869bc202201e06236a7272dada39e2bd66aa5c7/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 29 22:42:33.423886 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.422912 5057 manager.go:217] Machine: {Timestamp:2026-10-29 22:42:33.422447958 +0000 UTC m=+0.073946479 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099711 MemoryCapacity:33164320768 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c00acace8fc6bc6c7eede4040af50 SystemUUID:ec2c00ac-ace8-fc6b-c6c7-eede4040af50 BootID:f0329396-7fbc-4bfb-8f0c-8549fdfce190 Filesystems:[{Device:/var/lib/kubelet/pods/a1a232f4-f67c-404f-a1f7-af0c3cc30180/volumes/kubernetes.io~projected/kube-api-access-kgc6n DeviceMajor:0 DeviceMinor:219 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/877326a8-91a3-499f-8b37-4f80dfb00352/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:225 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/9483bf09-4676-47db-a279-84aabe9d17ed/volumes/kubernetes.io~projected/kube-api-access-w5d85 DeviceMajor:0 DeviceMinor:335 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6f91740-e20b-439e-be7d-4b444f4835fa/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:205 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99577b75-29cd-430b-b5ac-2af3828586d0/volumes/kubernetes.io~projected/kube-api-access-h744b DeviceMajor:0 DeviceMinor:226 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97fa3566-0c3f-4768-a437-3a89816303e0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/d6f91740-e20b-439e-be7d-4b444f4835fa/volumes/kubernetes.io~projected/kube-api-access-rx2rx DeviceMajor:0 DeviceMinor:227 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0e0c2bbab3f446ae654fa789a3fdca7a20acb188bbae830f145505f196b75fc/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/884370ff-c54f-4d6b-af78-5c0813bb48eb/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:215 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/463bc340-b1dc-41e6-9203-fcd2672f8a17/volumes/kubernetes.io~projected/kube-api-access-jk8rz DeviceMajor:0 DeviceMinor:313 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/093ceb90-294f-4133-83a6-32548405d530/volumes/kubernetes.io~projected/kube-api-access-5cffk DeviceMajor:0 DeviceMinor:212 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41ea529e-ab5d-419c-87bf-fe0bd1a6460f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:304 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99aaddab36725a8f6ffbbf5d65006c25fbd7bcca3de186b40c5774dbd8a614cb/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/710f6482706cf21d13a0be40293e5c922dc9549d6d94c8e010c661f794105f8c/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/73b41114-4ff0-4357-98b5-a871335ee3de/volumes/kubernetes.io~projected/kube-api-access-vfslj DeviceMajor:0 DeviceMinor:361 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a109715c-c29a-48d1-a573-3e7800f48259/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/a109715c-c29a-48d1-a573-3e7800f48259/volumes/kubernetes.io~projected/kube-api-access-zqxqz DeviceMajor:0 DeviceMinor:216 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a37a3da87cc440a67f1cefaefb01ac48e701dfc430b6bce0687dd5782f6833d7/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3629e1ac73011582ae081f05d21007b7c932456f98e64abbab89ce4fc161673/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/877326a8-91a3-499f-8b37-4f80dfb00352/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:206 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e0e85234849cda82191e957089de80d763789a100521253717a21405e99c869/userdata/shm DeviceMajor:0 DeviceMinor:319 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f59a804858118aa038ff14cb50db8dcd52f6a3704bb85fbcf18447550d056fe/userdata/shm DeviceMajor:0 DeviceMinor:323 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73b41114-4ff0-4357-98b5-a871335ee3de/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:360 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/af926a18-1147-4786-8bd9-8a6727abb6fa/volumes/kubernetes.io~projected/kube-api-access-2z9zh DeviceMajor:0 DeviceMinor:60 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/85325a00-d2ed-4c7d-b287-0fd5452f7e69/volumes/kubernetes.io~projected/kube-api-access-vrls7 DeviceMajor:0 DeviceMinor:62 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cea9c5c68a1c41e3617d64636095622a1097edea881f16f4fc74dadeea8e15e4/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b12aec2fedb8f3264afa0c6e4e36f3cc62676554e9adc0e336775257632dcf72/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71be8984-962e-4357-a596-b9d3931e3a75/volumes/kubernetes.io~projected/kube-api-access-rhmrs DeviceMajor:0 DeviceMinor:65 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71be8984-962e-4357-a596-b9d3931e3a75/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656/volumes/kubernetes.io~projected/kube-api-access-7hp7m DeviceMajor:0 DeviceMinor:61 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f94f8986e84080f4850d25025683022e39863ad901a59bfeb9eb02e89d5837da/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6a4e4c73-0516-464c-9b6e-1c4adf93c03b/volumes/kubernetes.io~projected/kube-api-access-dcjhj DeviceMajor:0 DeviceMinor:223 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ce7e52911f12103f3bf53cd734767cbff8b8fbdede8860e494cbcfc6f5813b7/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/ce5dd070-fce7-44da-aaeb-017109fc98c9/volumes/kubernetes.io~projected/kube-api-access-h4z7h DeviceMajor:0 DeviceMinor:334 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72c20a30bd389edb03b22011f445c4d40c2fa5a93e32a280e4278313668c7aca/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a289effe7cef40dca73995ac30e516b30174c991b7833f92e253ea116322a6d6/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/96726184-9be2-4685-93d1-415bbb8b551c/volumes/kubernetes.io~projected/kube-api-access-cg6m4 DeviceMajor:0 DeviceMinor:220 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582160384 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/4590699c-6eb3-4546-835f-924180048d8a/volumes/kubernetes.io~projected/kube-api-access-jdmbj DeviceMajor:0 DeviceMinor:64 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/286133c3-abe5-4954-83a1-8665c1785b92/volumes/kubernetes.io~projected/kube-api-access-qvrrh DeviceMajor:0 DeviceMinor:67 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82c4b5ea11fe463ac3c5b009a989c3434ee67cf2d2db2628c885902825057b10/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/884370ff-c54f-4d6b-af78-5c0813bb48eb/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/68daf26a-81a0-4bd5-a188-9b39a1f460a7/volumes/kubernetes.io~projected/kube-api-access-rz8wx DeviceMajor:0 DeviceMinor:214 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/71be8984-962e-4357-a596-b9d3931e3a75/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b4ef5a62c85870b52d54ad3ffd257dee8b1784d7fd5cbbf259f4e8855eb7afb/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/028b9bc723b018c49696c401214b8ec85c9b1f9d54c7f29a10ddfb3600d43b32/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74d538b607b308700c2ddf7b631b7a4f89045fef1002029b6d323b160552ac49/userdata/shm DeviceMajor:0 DeviceMinor:347 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/97fa3566-0c3f-4768-a437-3a89816303e0/volumes/kubernetes.io~projected/kube-api-access-plrh7 DeviceMajor:0 DeviceMinor:59 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4525bd80-124f-4894-9fc0-b8235a0fe689/volumes/kubernetes.io~projected/kube-api-access-gcb88 DeviceMajor:0 DeviceMinor:63 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/ad00084a-5a9f-489b-9122-2316f617f3f1/volumes/kubernetes.io~projected/kube-api-access-2kpn4 DeviceMajor:0 DeviceMinor:213 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99577b75-29cd-430b-b5ac-2af3828586d0/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:210 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/8f6c01b9-e044-413c-b533-09033d04f29e/volumes/kubernetes.io~projected/kube-api-access-m76zp DeviceMajor:0 DeviceMinor:221 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97fa3566-0c3f-4768-a437-3a89816303e0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/99577b75-29cd-430b-b5ac-2af3828586d0/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:197 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/85ab246f-f679-40c5-a127-c0fd5a4a621e/volumes/kubernetes.io~projected/kube-api-access-g9l7w DeviceMajor:0 DeviceMinor:211 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/266bff54-6027-4822-91ce-be5cbd316ddc/volumes/kubernetes.io~projected/kube-api-access-82r6j DeviceMajor:0 DeviceMinor:224 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9870c7f5e182878c16f0d6c428a1c62d407777effb29d920c0ffe042b0ed2873/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed05e90ad4316ad2bd053ae3fe901f90d0c3fe8f02ae86066c4e4e9f2729f8a6/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5148e4d802089db3f519fa75c16f6d11b1e8d116ca1337134eb4edf466cbacb6/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/884370ff-c54f-4d6b-af78-5c0813bb48eb/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:209 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac9dd7d8666ac55fac7d0ebe766e087035079f15b2f3dcffcfb734b1f0597a10/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e91a303-9b80-47f5-a502-fbb6365eaa54/volumes/kubernetes.io~projected/kube-api-access-zvm7r DeviceMajor:0 DeviceMinor:217 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8/volumes/kubernetes.io~projected/kube-api-access-f6nxm DeviceMajor:0 DeviceMinor:218 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd844cb3511c0f05c9f8c26c1c22d0494132ad245c74b1242938d0befbf75c70/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/877326a8-91a3-499f-8b37-4f80dfb00352/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:204 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/093ceb90-294f-4133-83a6-32548405d530/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9/volumes/kubernetes.io~projected/kube-api-access-jrhns DeviceMajor:0 DeviceMinor:314 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a/volumes/kubernetes.io~projected/kube-api-access-85sqm DeviceMajor:0 DeviceMinor:66 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/884370ff-c54f-4d6b-af78-5c0813bb48eb/volumes/kubernetes.io~projected/kube-api-access-t5l7q DeviceMajor:0 DeviceMinor:222 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/220631aa8bf08160ea2ae099ede645cd10b43b7f584b3f3f159920391e1cc22d/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/901f5e90bef358345098700155850e5133969b5fd52e69a23d1452675c671c23/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/877326a8-91a3-499f-8b37-4f80dfb00352/volumes/kubernetes.io~projected/kube-api-access-hcfvz DeviceMajor:0 DeviceMinor:228 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/c29d1f02-5697-4d94-8a21-0a39e8f473ed/volumes/kubernetes.io~projected/kube-api-access-cx7m8 DeviceMajor:0 DeviceMinor:328 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582160384 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e91a303-9b80-47f5-a502-fbb6365eaa54/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25d1297cee065faf2844f297078cbe34338e029f0120e4f6fbadc072a2e4f1e9/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e28f2f94763c91944b4b5e6fcdda5c2aa18ee62de01c0f5b3afbe0d2c67e887f/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:220631aa8bf0816 MacAddress:16:a3:20:19:10:1c Speed:10000 Mtu:8901} {Name:25d1297cee065fa MacAddress:36:7e:e7:91:f1:c3 Speed:10000 Mtu:8901} {Name:3ce7e52911f1210 MacAddress:7e:25:cc:8d:e9:d2 Speed:10000 Mtu:8901} {Name:3e0e85234849cda MacAddress:5e:c3:b0:a2:ca:53 Speed:10000 Mtu:8901} {Name:710f6482706cf21 MacAddress:16:49:37:d0:2e:48 Speed:10000 Mtu:8901} {Name:74d538b607b3087 MacAddress:76:f2:84:bc:c8:66 Speed:10000 Mtu:8901} {Name:8b4ef5a62c85870 MacAddress:b6:a6:f8:27:91:72 Speed:10000 Mtu:8901} {Name:9870c7f5e182878 MacAddress:96:90:04:ff:61:05 Speed:10000 Mtu:8901} {Name:99aaddab36725a8 MacAddress:3e:e2:17:0f:fc:09 Speed:10000 Mtu:8901} {Name:9f59a804858118a MacAddress:4e:3c:16:e5:a6:ea Speed:10000 Mtu:8901} {Name:a37a3da87cc440a MacAddress:de:85:06:1d:a4:63 Speed:10000 Mtu:8901} {Name:ac9dd7d8666ac55 MacAddress:02:f6:12:1a:91:28 Speed:10000 Mtu:8901} {Name:b0e0c2bbab3f446 MacAddress:62:fd:6e:3f:3a:91 Speed:10000 Mtu:8901} {Name:b12aec2fedb8f32 MacAddress:96:92:5f:c6:8c:56 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ff:c7:45:9b Speed:0 Mtu:9001} {Name:br-int MacAddress:e6:b7:64:7c:cb:77 Speed:0 Mtu:8901} {Name:e28f2f94763c919 MacAddress:96:42:f3:e0:41:17 Speed:10000 Mtu:8901} {Name:ed05e90ad4316ad MacAddress:7a:9a:7a:03:11:45 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ff:c7:45:9b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:5e:43:80:3b:26:24 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:d6:8c:a1:a7:15:01 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164320768 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 22:42:33.423886 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.423363 5057 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 22:42:33.423886 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.423411 5057 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 22:42:33.423886 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.423752 5057 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 22:42:33.424141 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.423771 5057 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-80.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 22:42:33.424141 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.423933 5057 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 22:42:33.424141 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.423941 5057 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 22:42:33.424141 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.423951 5057 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:42:33.424141 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.423967 5057 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:42:33.424141 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.424053 5057 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 22:42:33.424325 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.424262 5057 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 22:42:33.424413 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.424403 5057 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 22:42:33.424448 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.424440 5057 kubelet.go:493] "Attempting to sync node with API server" Oct 29 22:42:33.424476 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.424459 5057 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 22:42:33.424476 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.424468 5057 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 22:42:33.424549 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.424477 5057 kubelet.go:405] "Adding apiserver pod source" Oct 29 22:42:33.424549 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.424485 5057 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 22:42:33.426198 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.425830 5057 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 22:42:33.426198 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.425850 5057 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:42:33.426683 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.426570 5057 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 22:42:33.429281 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.429266 5057 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 22:42:33.429281 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.429280 5057 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:42:33.429394 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.429384 5057 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 29 22:42:33.429582 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.429571 5057 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 22:42:33.429615 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.429601 5057 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 22:42:33.429713 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.429704 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 22:42:33.429742 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.429719 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 22:42:33.429742 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.429726 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 22:42:33.429742 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.429732 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 22:42:33.429742 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.429742 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 22:42:33.429842 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.429751 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 22:42:33.429842 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.429761 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 22:42:33.429842 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.429768 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 22:42:33.429842 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.429775 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 22:42:33.429842 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.429781 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 22:42:33.429842 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.429795 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 22:42:33.429842 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.429805 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 22:42:33.429842 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.429832 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 22:42:33.429842 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.429841 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 22:42:33.433424 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.433407 5057 server.go:1263] "Started kubelet" Oct 29 22:42:33.433511 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.433457 5057 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 22:42:33.433618 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.433562 5057 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 22:42:33.433695 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.433680 5057 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 22:42:33.433986 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.433968 5057 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 22:42:33.434820 ip-10-0-135-80 systemd[1]: Started Kubernetes Kubelet. Oct 29 22:42:33.435684 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.435228 5057 server.go:317] "Adding debug handlers to kubelet server" Oct 29 22:42:33.437010 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.436989 5057 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 22:42:33.437124 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.437021 5057 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 22:42:33.437124 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.437074 5057 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:36:55 +0000 UTC" deadline="2028-07-23 03:39:35.292817846 +0000 UTC" Oct 29 22:42:33.437124 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.437103 5057 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15172h57m1.855717902s" Oct 29 22:42:33.440724 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:33.437137 5057 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-80.ec2.internal\" not found" Oct 29 22:42:33.440724 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.437958 5057 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 22:42:33.440724 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.438084 5057 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 22:42:33.440724 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.439220 5057 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 22:42:33.442713 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442560 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e91a303-9b80-47f5-a502-fbb6365eaa54" volumeName="kubernetes.io/projected/9e91a303-9b80-47f5-a502-fbb6365eaa54-kube-api-access-zvm7r" seLinuxMountContext="" Oct 29 22:42:33.442713 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442595 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c29d1f02-5697-4d94-8a21-0a39e8f473ed" volumeName="kubernetes.io/configmap/c29d1f02-5697-4d94-8a21-0a39e8f473ed-trusted-ca-bundle" seLinuxMountContext="" Oct 29 22:42:33.442713 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442609 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce5dd070-fce7-44da-aaeb-017109fc98c9" volumeName="kubernetes.io/projected/ce5dd070-fce7-44da-aaeb-017109fc98c9-kube-api-access-h4z7h" seLinuxMountContext="" Oct 29 22:42:33.442713 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442620 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="286133c3-abe5-4954-83a1-8665c1785b92" volumeName="kubernetes.io/projected/286133c3-abe5-4954-83a1-8665c1785b92-kube-api-access-qvrrh" seLinuxMountContext="" Oct 29 22:42:33.442713 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442633 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="877326a8-91a3-499f-8b37-4f80dfb00352" volumeName="kubernetes.io/secret/877326a8-91a3-499f-8b37-4f80dfb00352-installation-pull-secrets" seLinuxMountContext="" Oct 29 22:42:33.442713 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442643 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d2cdca1-7303-4d6c-98ea-5eb6b7d16656" volumeName="kubernetes.io/projected/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656-kube-api-access-7hp7m" seLinuxMountContext="" Oct 29 22:42:33.442713 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442655 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9483bf09-4676-47db-a279-84aabe9d17ed" volumeName="kubernetes.io/projected/9483bf09-4676-47db-a279-84aabe9d17ed-kube-api-access-w5d85" seLinuxMountContext="" Oct 29 22:42:33.442713 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442666 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d518d30f-c46b-49ca-b74c-9f4cf0ca71e9" volumeName="kubernetes.io/projected/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-api-access-jrhns" seLinuxMountContext="" Oct 29 22:42:33.442713 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442682 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="877326a8-91a3-499f-8b37-4f80dfb00352" volumeName="kubernetes.io/empty-dir/877326a8-91a3-499f-8b37-4f80dfb00352-ca-trust-extracted" seLinuxMountContext="" Oct 29 22:42:33.442713 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442696 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884370ff-c54f-4d6b-af78-5c0813bb48eb" volumeName="kubernetes.io/empty-dir/884370ff-c54f-4d6b-af78-5c0813bb48eb-ca-trust-extracted" seLinuxMountContext="" Oct 29 22:42:33.442713 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442712 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af926a18-1147-4786-8bd9-8a6727abb6fa" volumeName="kubernetes.io/projected/af926a18-1147-4786-8bd9-8a6727abb6fa-kube-api-access-2z9zh" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442725 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c29d1f02-5697-4d94-8a21-0a39e8f473ed" volumeName="kubernetes.io/configmap/c29d1f02-5697-4d94-8a21-0a39e8f473ed-service-ca-bundle" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442738 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c29d1f02-5697-4d94-8a21-0a39e8f473ed" volumeName="kubernetes.io/empty-dir/c29d1f02-5697-4d94-8a21-0a39e8f473ed-archives-path" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442750 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="575ee3c4-7c77-4b31-9961-eebc60254d19" volumeName="kubernetes.io/configmap/575ee3c4-7c77-4b31-9961-eebc60254d19-nginx-conf" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442767 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68daf26a-81a0-4bd5-a188-9b39a1f460a7" volumeName="kubernetes.io/projected/68daf26a-81a0-4bd5-a188-9b39a1f460a7-kube-api-access-rz8wx" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442781 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="877326a8-91a3-499f-8b37-4f80dfb00352" volumeName="kubernetes.io/configmap/877326a8-91a3-499f-8b37-4f80dfb00352-registry-certificates" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442796 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6f91740-e20b-439e-be7d-4b444f4835fa" volumeName="kubernetes.io/projected/d6f91740-e20b-439e-be7d-4b444f4835fa-kube-api-access-rx2rx" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442809 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884370ff-c54f-4d6b-af78-5c0813bb48eb" volumeName="kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-bound-sa-token" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442822 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="266bff54-6027-4822-91ce-be5cbd316ddc" volumeName="kubernetes.io/configmap/266bff54-6027-4822-91ce-be5cbd316ddc-config-volume" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442834 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4525bd80-124f-4894-9fc0-b8235a0fe689" volumeName="kubernetes.io/configmap/4525bd80-124f-4894-9fc0-b8235a0fe689-serviceca" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442847 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71be8984-962e-4357-a596-b9d3931e3a75" volumeName="kubernetes.io/projected/71be8984-962e-4357-a596-b9d3931e3a75-kube-api-access-rhmrs" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442858 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884370ff-c54f-4d6b-af78-5c0813bb48eb" volumeName="kubernetes.io/configmap/884370ff-c54f-4d6b-af78-5c0813bb48eb-trusted-ca" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442872 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a109715c-c29a-48d1-a573-3e7800f48259" volumeName="kubernetes.io/secret/a109715c-c29a-48d1-a573-3e7800f48259-serving-cert" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442880 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8" volumeName="kubernetes.io/configmap/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-service-ca-bundle" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442888 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c29d1f02-5697-4d94-8a21-0a39e8f473ed" volumeName="kubernetes.io/projected/c29d1f02-5697-4d94-8a21-0a39e8f473ed-kube-api-access-cx7m8" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442895 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="266bff54-6027-4822-91ce-be5cbd316ddc" volumeName="kubernetes.io/empty-dir/266bff54-6027-4822-91ce-be5cbd316ddc-tmp-dir" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442910 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9" volumeName="kubernetes.io/configmap/2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9-konnectivity-ca" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442918 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4590699c-6eb3-4546-835f-924180048d8a" volumeName="kubernetes.io/projected/4590699c-6eb3-4546-835f-924180048d8a-kube-api-access-jdmbj" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442928 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73b41114-4ff0-4357-98b5-a871335ee3de" volumeName="kubernetes.io/projected/73b41114-4ff0-4357-98b5-a871335ee3de-kube-api-access-vfslj" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442936 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85325a00-d2ed-4c7d-b287-0fd5452f7e69" volumeName="kubernetes.io/configmap/85325a00-d2ed-4c7d-b287-0fd5452f7e69-cni-binary-copy" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442944 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99577b75-29cd-430b-b5ac-2af3828586d0" volumeName="kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-default-certificate" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442956 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="266bff54-6027-4822-91ce-be5cbd316ddc" volumeName="kubernetes.io/projected/266bff54-6027-4822-91ce-be5cbd316ddc-kube-api-access-82r6j" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442964 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="463bc340-b1dc-41e6-9203-fcd2672f8a17" volumeName="kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442972 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d2cdca1-7303-4d6c-98ea-5eb6b7d16656" volumeName="kubernetes.io/configmap/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656-iptables-alerter-script" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442981 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a109715c-c29a-48d1-a573-3e7800f48259" volumeName="kubernetes.io/projected/a109715c-c29a-48d1-a573-3e7800f48259-kube-api-access-zqxqz" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.442997 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad00084a-5a9f-489b-9122-2316f617f3f1" volumeName="kubernetes.io/configmap/ad00084a-5a9f-489b-9122-2316f617f3f1-telemetry-config" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443004 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af926a18-1147-4786-8bd9-8a6727abb6fa" volumeName="kubernetes.io/empty-dir/af926a18-1147-4786-8bd9-8a6727abb6fa-tmp-dir" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443011 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="093ceb90-294f-4133-83a6-32548405d530" volumeName="kubernetes.io/configmap/093ceb90-294f-4133-83a6-32548405d530-config" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443019 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2863700d-26b0-4e9d-bd6a-ce8939cf3c9a" volumeName="kubernetes.io/projected/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-kube-api-access-85sqm" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443030 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71be8984-962e-4357-a596-b9d3931e3a75" volumeName="kubernetes.io/configmap/71be8984-962e-4357-a596-b9d3931e3a75-env-overrides" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443041 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85325a00-d2ed-4c7d-b287-0fd5452f7e69" volumeName="kubernetes.io/configmap/85325a00-d2ed-4c7d-b287-0fd5452f7e69-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443051 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="877326a8-91a3-499f-8b37-4f80dfb00352" volumeName="kubernetes.io/secret/877326a8-91a3-499f-8b37-4f80dfb00352-image-registry-private-configuration" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443059 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884370ff-c54f-4d6b-af78-5c0813bb48eb" volumeName="kubernetes.io/secret/884370ff-c54f-4d6b-af78-5c0813bb48eb-installation-pull-secrets" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443067 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97fa3566-0c3f-4768-a437-3a89816303e0" volumeName="kubernetes.io/projected/97fa3566-0c3f-4768-a437-3a89816303e0-kube-api-access-plrh7" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443075 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e91a303-9b80-47f5-a502-fbb6365eaa54" volumeName="kubernetes.io/secret/9e91a303-9b80-47f5-a502-fbb6365eaa54-serving-cert" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443084 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85325a00-d2ed-4c7d-b287-0fd5452f7e69" volumeName="kubernetes.io/projected/85325a00-d2ed-4c7d-b287-0fd5452f7e69-kube-api-access-vrls7" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443097 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884370ff-c54f-4d6b-af78-5c0813bb48eb" volumeName="kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-kube-api-access-t5l7q" seLinuxMountContext="" Oct 29 22:42:33.443082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443107 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96726184-9be2-4685-93d1-415bbb8b551c" volumeName="kubernetes.io/projected/96726184-9be2-4685-93d1-415bbb8b551c-kube-api-access-cg6m4" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443115 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99577b75-29cd-430b-b5ac-2af3828586d0" volumeName="kubernetes.io/projected/99577b75-29cd-430b-b5ac-2af3828586d0-kube-api-access-h744b" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443123 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99577b75-29cd-430b-b5ac-2af3828586d0" volumeName="kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-stats-auth" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443131 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a109715c-c29a-48d1-a573-3e7800f48259" volumeName="kubernetes.io/configmap/a109715c-c29a-48d1-a573-3e7800f48259-config" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443138 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8" volumeName="kubernetes.io/configmap/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-trusted-ca-bundle" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443148 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8" volumeName="kubernetes.io/empty-dir/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-tmp" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443159 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6f91740-e20b-439e-be7d-4b444f4835fa" volumeName="kubernetes.io/secret/d6f91740-e20b-439e-be7d-4b444f4835fa-cert" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443171 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="093ceb90-294f-4133-83a6-32548405d530" volumeName="kubernetes.io/projected/093ceb90-294f-4133-83a6-32548405d530-kube-api-access-5cffk" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443180 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2863700d-26b0-4e9d-bd6a-ce8939cf3c9a" volumeName="kubernetes.io/configmap/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-cni-binary-copy" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443211 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a4e4c73-0516-464c-9b6e-1c4adf93c03b" volumeName="kubernetes.io/projected/6a4e4c73-0516-464c-9b6e-1c4adf93c03b-kube-api-access-dcjhj" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443222 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73b41114-4ff0-4357-98b5-a871335ee3de" volumeName="kubernetes.io/secret/73b41114-4ff0-4357-98b5-a871335ee3de-signing-key" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443231 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="877326a8-91a3-499f-8b37-4f80dfb00352" volumeName="kubernetes.io/configmap/877326a8-91a3-499f-8b37-4f80dfb00352-trusted-ca" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443238 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884370ff-c54f-4d6b-af78-5c0813bb48eb" volumeName="kubernetes.io/secret/884370ff-c54f-4d6b-af78-5c0813bb48eb-image-registry-private-configuration" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443245 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96726184-9be2-4685-93d1-415bbb8b551c" volumeName="kubernetes.io/configmap/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-config" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443252 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e91a303-9b80-47f5-a502-fbb6365eaa54" volumeName="kubernetes.io/configmap/9e91a303-9b80-47f5-a502-fbb6365eaa54-config" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443260 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9" volumeName="kubernetes.io/secret/2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9-agent-certs" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443268 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884370ff-c54f-4d6b-af78-5c0813bb48eb" volumeName="kubernetes.io/configmap/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-certificates" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443281 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e91a303-9b80-47f5-a502-fbb6365eaa54" volumeName="kubernetes.io/configmap/9e91a303-9b80-47f5-a502-fbb6365eaa54-trusted-ca" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443289 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8" volumeName="kubernetes.io/empty-dir/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-snapshots" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443305 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d518d30f-c46b-49ca-b74c-9f4cf0ca71e9" volumeName="kubernetes.io/empty-dir/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-data-volume" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443313 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="093ceb90-294f-4133-83a6-32548405d530" volumeName="kubernetes.io/secret/093ceb90-294f-4133-83a6-32548405d530-serving-cert" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443321 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2863700d-26b0-4e9d-bd6a-ce8939cf3c9a" volumeName="kubernetes.io/configmap/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-daemon-config" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443328 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71be8984-962e-4357-a596-b9d3931e3a75" volumeName="kubernetes.io/configmap/71be8984-962e-4357-a596-b9d3931e3a75-ovnkube-script-lib" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443336 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71be8984-962e-4357-a596-b9d3931e3a75" volumeName="kubernetes.io/secret/71be8984-962e-4357-a596-b9d3931e3a75-ovn-node-metrics-cert" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443344 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="877326a8-91a3-499f-8b37-4f80dfb00352" volumeName="kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-bound-sa-token" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443351 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="877326a8-91a3-499f-8b37-4f80dfb00352" volumeName="kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-kube-api-access-hcfvz" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443359 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1a232f4-f67c-404f-a1f7-af0c3cc30180" volumeName="kubernetes.io/projected/a1a232f4-f67c-404f-a1f7-af0c3cc30180-kube-api-access-kgc6n" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443368 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f6c01b9-e044-413c-b533-09033d04f29e" volumeName="kubernetes.io/projected/8f6c01b9-e044-413c-b533-09033d04f29e-kube-api-access-m76zp" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443375 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97fa3566-0c3f-4768-a437-3a89816303e0" volumeName="kubernetes.io/empty-dir/97fa3566-0c3f-4768-a437-3a89816303e0-tmp" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443383 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85ab246f-f679-40c5-a127-c0fd5a4a621e" volumeName="kubernetes.io/projected/85ab246f-f679-40c5-a127-c0fd5a4a621e-kube-api-access-g9l7w" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443391 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8" volumeName="kubernetes.io/projected/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-kube-api-access-f6nxm" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443398 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad00084a-5a9f-489b-9122-2316f617f3f1" volumeName="kubernetes.io/projected/ad00084a-5a9f-489b-9122-2316f617f3f1-kube-api-access-2kpn4" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443405 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4525bd80-124f-4894-9fc0-b8235a0fe689" volumeName="kubernetes.io/projected/4525bd80-124f-4894-9fc0-b8235a0fe689-kube-api-access-gcb88" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443412 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71be8984-962e-4357-a596-b9d3931e3a75" volumeName="kubernetes.io/configmap/71be8984-962e-4357-a596-b9d3931e3a75-ovnkube-config" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443420 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85325a00-d2ed-4c7d-b287-0fd5452f7e69" volumeName="kubernetes.io/configmap/85325a00-d2ed-4c7d-b287-0fd5452f7e69-cni-sysctl-allowlist" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443428 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8" volumeName="kubernetes.io/secret/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-serving-cert" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443357 5057 factory.go:55] Registering systemd factory Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443448 5057 factory.go:223] Registration of the systemd container factory successfully Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443435 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41ea529e-ab5d-419c-87bf-fe0bd1a6460f" volumeName="kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443509 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73b41114-4ff0-4357-98b5-a871335ee3de" volumeName="kubernetes.io/configmap/73b41114-4ff0-4357-98b5-a871335ee3de-signing-cabundle" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443530 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97fa3566-0c3f-4768-a437-3a89816303e0" volumeName="kubernetes.io/empty-dir/97fa3566-0c3f-4768-a437-3a89816303e0-etc-tuned" seLinuxMountContext="" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443543 5057 reconstruct.go:97] "Volume reconstruction finished" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443550 5057 reconciler.go:29] "Reconciler: start to sync state" Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443921 5057 factory.go:153] Registering CRI-O factory Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443937 5057 factory.go:223] Registration of the crio container factory successfully Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.443986 5057 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.444011 5057 factory.go:103] Registering Raw factory Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.444025 5057 manager.go:1199] Started watching for new ooms in manager Oct 29 22:42:33.444670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.444416 5057 manager.go:319] Starting recovery of all containers Oct 29 22:42:33.463913 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.463888 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 22:42:33.464555 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.464518 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 22:42:33.470882 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.470838 5057 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 22:42:33.472716 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.472622 5057 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 22:42:33.472716 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.472651 5057 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 22:42:33.472716 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.472672 5057 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 22:42:33.472716 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:33.472710 5057 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 29 22:42:33.488700 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.488677 5057 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 29 22:42:33.488877 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.488858 5057 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:42:33.497635 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.497613 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 22:42:33.497710 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.497647 5057 generic.go:356] "Generic (PLEG): container finished" podID="9e91a303-9b80-47f5-a502-fbb6365eaa54" containerID="f0fcd8c76c7857be846eac042de3c1bf7b719752c31262e9fa639c35cfdc6aac" exitCode=255 Oct 29 22:42:33.500516 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.499517 5057 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:42:33.502118 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.502100 5057 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:42:33.502365 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.502349 5057 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:42:33.511616 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.511586 5057 generic.go:356] "Generic (PLEG): container finished" podID="85325a00-d2ed-4c7d-b287-0fd5452f7e69" containerID="6240c2e598a9c0ce9990608cd183149def9de5756569d7b596be7796c6d9a55e" exitCode=0 Oct 29 22:42:33.511616 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.511615 5057 generic.go:356] "Generic (PLEG): container finished" podID="85325a00-d2ed-4c7d-b287-0fd5452f7e69" containerID="8966cb9af80afb5802f7a7ecf814af8a3c9943662aca26aa159e285ae3e94f82" exitCode=0 Oct 29 22:42:33.511726 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.511626 5057 generic.go:356] "Generic (PLEG): container finished" podID="85325a00-d2ed-4c7d-b287-0fd5452f7e69" containerID="7525ecf3c4fb8cd0826fe365d3519374edf095e2c3c378cc5a1c5939168df5a5" exitCode=0 Oct 29 22:42:33.511726 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.511636 5057 generic.go:356] "Generic (PLEG): container finished" podID="85325a00-d2ed-4c7d-b287-0fd5452f7e69" containerID="28dc5d6af575caecd2f55645a5d1a9593e92ab730789225034ce54f0eaa2cc56" exitCode=0 Oct 29 22:42:33.511726 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.511645 5057 generic.go:356] "Generic (PLEG): container finished" podID="85325a00-d2ed-4c7d-b287-0fd5452f7e69" containerID="7912f888f1d0da0c4e179a31895cf9c9bacaa8d47d432fb4131bc6e2746498d2" exitCode=0 Oct 29 22:42:33.511726 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.511654 5057 generic.go:356] "Generic (PLEG): container finished" podID="85325a00-d2ed-4c7d-b287-0fd5452f7e69" containerID="e96d06cb90b7322155ae1238e68bd7752d79d6ac6863c4a4a461dfb04c204660" exitCode=0 Oct 29 22:42:33.517953 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.517929 5057 generic.go:356] "Generic (PLEG): container finished" podID="b94a7f102f3e9c0fab9ce63f581b4ba1" containerID="feafcd201853adb3f768525d57f4e273d0fb25ac84315031e3df0a74c5419c57" exitCode=0 Oct 29 22:42:33.542142 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.542086 5057 manager.go:324] Recovery completed Oct 29 22:42:33.561993 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.561965 5057 cpu_manager.go:226] "Starting" policy="none" Oct 29 22:42:33.561993 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.561989 5057 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 22:42:33.562168 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.562011 5057 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 22:42:33.562287 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.562215 5057 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 29 22:42:33.562287 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.562230 5057 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 29 22:42:33.562287 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.562262 5057 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 29 22:42:33.562287 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.562270 5057 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 29 22:42:33.562287 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.562278 5057 policy_none.go:50] "Start" Oct 29 22:42:33.562518 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.562291 5057 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 22:42:33.562518 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.562309 5057 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 22:42:33.562603 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.562532 5057 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 29 22:42:33.562603 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.562543 5057 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 29 22:42:33.562603 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.562551 5057 policy_none.go:44] "Start" Oct 29 22:42:33.567477 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.567448 5057 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 22:42:33.570088 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.570076 5057 manager.go:341] "Starting Device Plugin manager" Oct 29 22:42:33.570792 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:33.570101 5057 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 22:42:33.570863 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.570796 5057 server.go:87] "Starting device plugin registration server" Oct 29 22:42:33.571123 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.571112 5057 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 22:42:33.571169 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.571126 5057 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 22:42:33.571269 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.571251 5057 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 22:42:33.571386 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.571356 5057 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 22:42:33.571450 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.571394 5057 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 22:42:33.571450 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.571401 5057 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 22:42:33.571536 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.571452 5057 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:42:33.571383608Z","UUID":"ecc09a7c-3e16-47ad-a2be-9dff847a0d0b","Handler":null,"Name":"","Endpoint":""} Oct 29 22:42:33.572714 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.572682 5057 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 22:42:33.572801 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.572716 5057 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 22:42:33.572844 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:33.572821 5057 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 22:42:33.573100 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.573066 5057 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal"] Oct 29 22:42:33.574151 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.573841 5057 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" Oct 29 22:42:33.574151 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.573976 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" event={"ID":"b94a7f102f3e9c0fab9ce63f581b4ba1","Type":"ContainerStarted","Data":"a2ccb0138cf51229483fc46416519e08e032de50f25307865d6e63a6a4cc0fd1"} Oct 29 22:42:33.574151 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.574028 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" event={"ID":"b94a7f102f3e9c0fab9ce63f581b4ba1","Type":"ContainerDied","Data":"feafcd201853adb3f768525d57f4e273d0fb25ac84315031e3df0a74c5419c57"} Oct 29 22:42:33.574151 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.574049 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" event={"ID":"b94a7f102f3e9c0fab9ce63f581b4ba1","Type":"ContainerStarted","Data":"72c20a30bd389edb03b22011f445c4d40c2fa5a93e32a280e4278313668c7aca"} Oct 29 22:42:33.574151 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.574067 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" event={"ID":"4239c6df062516f41cb9c5984a7633c8","Type":"ContainerStarted","Data":"71d1e51ebf55437eece7c4675c9d51959adc116f991ae36a042ba715a13e0355"} Oct 29 22:42:33.574151 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.574086 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" event={"ID":"4239c6df062516f41cb9c5984a7633c8","Type":"ContainerStarted","Data":"901f5e90bef358345098700155850e5133969b5fd52e69a23d1452675c671c23"} Oct 29 22:42:33.574932 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.574912 5057 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Oct 29 22:42:33.595012 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.594993 5057 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:42:33.595102 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:33.595032 5057 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-80.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" Oct 29 22:42:33.604384 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.604360 5057 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:42:33.604478 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:33.604400 5057 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Oct 29 22:42:33.672255 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.672226 5057 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:42:33.673517 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.673499 5057 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:42:33.673627 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.673530 5057 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:42:33.673627 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.673545 5057 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:42:33.673627 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.673602 5057 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-80.ec2.internal" Oct 29 22:42:33.689592 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.689569 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b94a7f102f3e9c0fab9ce63f581b4ba1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal\" (UID: \"b94a7f102f3e9c0fab9ce63f581b4ba1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Oct 29 22:42:33.689664 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.689601 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b94a7f102f3e9c0fab9ce63f581b4ba1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal\" (UID: \"b94a7f102f3e9c0fab9ce63f581b4ba1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Oct 29 22:42:33.689664 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.689624 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4239c6df062516f41cb9c5984a7633c8-config\") pod \"kube-apiserver-proxy-ip-10-0-135-80.ec2.internal\" (UID: \"4239c6df062516f41cb9c5984a7633c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" Oct 29 22:42:33.710054 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.710029 5057 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-80.ec2.internal" Oct 29 22:42:33.710141 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.710098 5057 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-80.ec2.internal" Oct 29 22:42:33.789958 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.789920 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b94a7f102f3e9c0fab9ce63f581b4ba1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal\" (UID: \"b94a7f102f3e9c0fab9ce63f581b4ba1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Oct 29 22:42:33.790111 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.789965 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b94a7f102f3e9c0fab9ce63f581b4ba1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal\" (UID: \"b94a7f102f3e9c0fab9ce63f581b4ba1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Oct 29 22:42:33.790111 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.789992 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4239c6df062516f41cb9c5984a7633c8-config\") pod \"kube-apiserver-proxy-ip-10-0-135-80.ec2.internal\" (UID: \"4239c6df062516f41cb9c5984a7633c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" Oct 29 22:42:33.790111 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.790007 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b94a7f102f3e9c0fab9ce63f581b4ba1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal\" (UID: \"b94a7f102f3e9c0fab9ce63f581b4ba1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Oct 29 22:42:33.790111 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.790040 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4239c6df062516f41cb9c5984a7633c8-config\") pod \"kube-apiserver-proxy-ip-10-0-135-80.ec2.internal\" (UID: \"4239c6df062516f41cb9c5984a7633c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" Oct 29 22:42:33.790111 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:33.790084 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b94a7f102f3e9c0fab9ce63f581b4ba1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal\" (UID: \"b94a7f102f3e9c0fab9ce63f581b4ba1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Oct 29 22:42:34.425458 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.425400 5057 apiserver.go:52] "Watching apiserver" Oct 29 22:42:34.480163 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.480131 5057 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 22:42:34.480835 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.480811 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-868dt","kube-system/konnectivity-agent-sc55n","openshift-kube-storage-version-migrator/migrator-66df5d496b-n56zr","kube-system/kas-connection-checker-685895c7c9-7hz4v","openshift-console-operator/console-operator-7cd56b8dbc-lm55w","openshift-image-registry/image-registry-5cf56cb474-gx24x","kube-system/global-pull-secret-syncer-b2cf2","openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b","openshift-image-registry/node-ca-rwz4x","kube-system/kas-connection-checker-685895c7c9-fh7rx","openshift-ingress/router-default-7945df5b5-pcgp6","openshift-ingress-canary/ingress-canary-qbv9d","openshift-insights/periodic-gathering-vqgrg-xtvww","kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal","openshift-dns/node-resolver-5czm4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal","openshift-multus/multus-additional-cni-plugins-vvp9h","openshift-network-diagnostics/network-check-target-j8fft","openshift-image-registry/image-registry-78f4dff4c9-87jff","openshift-network-console/networking-console-plugin-86c478bb88-rwfmh","openshift-service-ca/service-ca-5fffc97bdf-d49cl","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh","openshift-insights/insights-runtime-extractor-6rqj6","openshift-multus/network-metrics-daemon-mxg8q","openshift-cluster-node-tuning-operator/tuned-flzmd","openshift-ovn-kubernetes/ovnkube-node-99ngw","openshift-multus/multus-qqk5c","openshift-dns/dns-default-gj6vs","openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf","openshift-network-operator/iptables-alerter-trj6w","openshift-kube-storage-version-migrator/migrator-66df5d496b-zl99j","openshift-insights/insights-operator-7c4986d449-9qhv9","openshift-network-diagnostics/network-check-source-6f78d59f99-kw5c9","kube-system/kas-connection-checker-685895c7c9-k8xnq"] Oct 29 22:42:34.483222 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.483198 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:34.483931 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.483911 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:34.484396 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.484377 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:34.485172 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.485157 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:34.485546 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.485528 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-rwfmh" Oct 29 22:42:34.496397 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.496368 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:34.497092 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.497069 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:34.497580 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.497564 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 29 22:42:34.497671 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.497618 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 29 22:42:34.497671 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.497634 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 29 22:42:34.497847 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.497835 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:34.498578 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.498549 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6l2xz\"" Oct 29 22:42:34.498578 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.498556 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 22:42:34.498722 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.498636 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-h9lx2\"" Oct 29 22:42:34.498722 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.498697 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nqrw2\"" Oct 29 22:42:34.498922 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.498902 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 29 22:42:34.499673 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.499657 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 22:42:34.499752 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.499734 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 29 22:42:34.499795 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.499755 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 29 22:42:34.500113 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.500100 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:34.500690 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.500673 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:42:34.500788 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.500705 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 29 22:42:34.500788 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.500713 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 29 22:42:34.500916 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.500724 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-m9r9w\"" Oct 29 22:42:34.508711 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.508692 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 29 22:42:34.511261 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.511248 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 29 22:42:34.511326 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.511271 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 22:42:34.511395 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.511372 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:42:34.511465 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.511424 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 22:42:34.511465 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.511426 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 29 22:42:34.511465 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.511451 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 29 22:42:34.511862 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.511826 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 29 22:42:34.511862 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.511826 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 29 22:42:34.511988 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.511876 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 29 22:42:34.511988 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.511928 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:42:34.511988 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.511938 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 29 22:42:34.511988 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.511963 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 22:42:34.511988 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.511970 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 29 22:42:34.512223 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.512200 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 29 22:42:34.512311 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.512287 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 29 22:42:34.512369 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.512359 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 29 22:42:34.512807 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.512790 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 29 22:42:34.524428 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.524407 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:42:34.526827 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.526805 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6m48j\"" Oct 29 22:42:34.529216 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.529180 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 22:42:34.529311 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.529258 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 22:42:34.529311 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.529304 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 22:42:34.529455 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.529440 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 22:42:34.529455 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.529446 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 22:42:34.529581 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.529568 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 22:42:34.529771 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.529758 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 29 22:42:34.529834 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.529801 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 29 22:42:34.529887 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.529873 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 29 22:42:34.529931 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.529903 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 22:42:34.530813 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.530795 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 29 22:42:34.530896 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.530882 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 22:42:34.531002 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.530990 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d6hqb\"" Oct 29 22:42:34.531092 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.531080 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 22:42:34.531145 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.531120 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 22:42:34.531286 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.531268 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tj892\"" Oct 29 22:42:34.531406 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.531294 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 22:42:34.531535 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.531370 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-28brx\"" Oct 29 22:42:34.531647 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.531536 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:42:34.531786 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.531767 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 29 22:42:34.531913 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.531821 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 22:42:34.531913 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.531775 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 22:42:34.531913 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.531889 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 29 22:42:34.531913 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.531905 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-gpfcv\"" Oct 29 22:42:34.532334 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532062 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-cz727\"" Oct 29 22:42:34.532334 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532078 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m52c5\"" Oct 29 22:42:34.532334 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532204 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 22:42:34.532334 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532214 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 29 22:42:34.532334 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532323 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 29 22:42:34.532610 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532326 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 29 22:42:34.532610 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532280 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vscck\"" Oct 29 22:42:34.532610 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532375 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-p26v7\"" Oct 29 22:42:34.532610 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532386 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-fchfg\"" Oct 29 22:42:34.532610 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532409 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 22:42:34.532610 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532439 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 29 22:42:34.532610 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532505 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 29 22:42:34.532610 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532526 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6752d\"" Oct 29 22:42:34.532610 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532571 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 22:42:34.532610 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532582 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 22:42:34.533001 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532601 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ltz9s\"" Oct 29 22:42:34.533001 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532665 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 22:42:34.533001 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532754 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 29 22:42:34.533001 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532797 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 22:42:34.533001 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532823 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:42:34.533001 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.532850 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 22:42:34.533271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533237 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 22:42:34.533271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533248 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 22:42:34.533271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533263 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 22:42:34.533470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533291 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 29 22:42:34.533470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533292 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 22:42:34.533470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533312 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 22:42:34.533470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533321 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 29 22:42:34.533470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533359 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:42:34.533470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533366 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 22:42:34.533470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533394 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 22:42:34.533470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533397 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-phcjr\"" Oct 29 22:42:34.533470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533392 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 22:42:34.533470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533424 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 22:42:34.533470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533440 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 29 22:42:34.533470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533445 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 22:42:34.533470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533454 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 29 22:42:34.533470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533461 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 22:42:34.533470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533466 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 29 22:42:34.534140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533486 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n7jjw\"" Oct 29 22:42:34.534140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533529 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-65zqz\"" Oct 29 22:42:34.534140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533558 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 22:42:34.534140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533596 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rpfjl\"" Oct 29 22:42:34.534140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533616 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 29 22:42:34.534140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533645 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h2plp\"" Oct 29 22:42:34.534140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533673 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mcv8v\"" Oct 29 22:42:34.534140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533718 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kfq4\"" Oct 29 22:42:34.534140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533777 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 29 22:42:34.534140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533863 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nxz4d\"" Oct 29 22:42:34.534140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.533940 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-rzm2m\"" Oct 29 22:42:34.534140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.534013 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xjmd7\"" Oct 29 22:42:34.534140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.534083 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-n2tqz\"" Oct 29 22:42:34.534140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.534128 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 22:42:34.534854 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.534552 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 29 22:42:34.535615 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.535600 5057 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Oct 29 22:42:34.535721 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.535709 5057 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" Oct 29 22:42:34.541630 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.541615 5057 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 22:42:34.564194 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.564163 5057 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:42:34.564290 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:34.564230 5057 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-80.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" Oct 29 22:42:34.564535 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.564522 5057 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:42:34.564575 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:34.564555 5057 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Oct 29 22:42:34.586910 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.586889 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rwz4x_4525bd80-124f-4894-9fc0-b8235a0fe689/node-ca/0.log" Oct 29 22:42:34.593239 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593214 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-cni-binary-copy\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.593314 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593252 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-var-lib-cni-bin\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.593314 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593282 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85325a00-d2ed-4c7d-b287-0fd5452f7e69-cnibin\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:42:34.593409 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593352 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-snapshots\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:34.593409 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593380 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8ncgh\" (UID: \"96726184-9be2-4685-93d1-415bbb8b551c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:34.593409 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593403 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/877326a8-91a3-499f-8b37-4f80dfb00352-ca-trust-extracted\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:34.593517 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593422 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-conf-dir\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.593517 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593446 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85sqm\" (UniqueName: \"kubernetes.io/projected/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-kube-api-access-85sqm\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.593517 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593479 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-node-log\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.593650 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593534 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/877326a8-91a3-499f-8b37-4f80dfb00352-ca-trust-extracted\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:34.593650 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593550 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85325a00-d2ed-4c7d-b287-0fd5452f7e69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:42:34.593650 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593565 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-snapshots\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:34.593650 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593602 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvm7r\" (UniqueName: \"kubernetes.io/projected/9e91a303-9b80-47f5-a502-fbb6365eaa54-kube-api-access-zvm7r\") pod \"console-operator-7cd56b8dbc-lm55w\" (UID: \"9e91a303-9b80-47f5-a502-fbb6365eaa54\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:34.593650 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593629 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-cni-binary-copy\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.593650 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593630 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-sys\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.593936 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593670 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrhns\" (UniqueName: \"kubernetes.io/projected/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-api-access-jrhns\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:34.593936 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593697 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656-iptables-alerter-script\") pod \"iptables-alerter-trj6w\" (UID: \"8d2cdca1-7303-4d6c-98ea-5eb6b7d16656\") " pod="openshift-network-operator/iptables-alerter-trj6w" Oct 29 22:42:34.593936 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593737 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-cni-bin\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.593936 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593761 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71be8984-962e-4357-a596-b9d3931e3a75-ovnkube-config\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.593936 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593785 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/877326a8-91a3-499f-8b37-4f80dfb00352-trusted-ca\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:34.593936 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593828 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-cnibin\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.593936 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593855 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-systemd\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.594227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593966 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af926a18-1147-4786-8bd9-8a6727abb6fa-tmp-dir\") pod \"node-resolver-5czm4\" (UID: \"af926a18-1147-4786-8bd9-8a6727abb6fa\") " pod="openshift-dns/node-resolver-5czm4" Oct 29 22:42:34.594227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.593999 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656-iptables-alerter-script\") pod \"iptables-alerter-trj6w\" (UID: \"8d2cdca1-7303-4d6c-98ea-5eb6b7d16656\") " pod="openshift-network-operator/iptables-alerter-trj6w" Oct 29 22:42:34.594227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594033 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-system-cni-dir\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.594227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594042 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af926a18-1147-4786-8bd9-8a6727abb6fa-tmp-dir\") pod \"node-resolver-5czm4\" (UID: \"af926a18-1147-4786-8bd9-8a6727abb6fa\") " pod="openshift-dns/node-resolver-5czm4" Oct 29 22:42:34.594227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594052 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/73b41114-4ff0-4357-98b5-a871335ee3de-signing-key\") pod \"service-ca-5fffc97bdf-d49cl\" (UID: \"73b41114-4ff0-4357-98b5-a871335ee3de\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d49cl" Oct 29 22:42:34.594227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594084 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfslj\" (UniqueName: \"kubernetes.io/projected/73b41114-4ff0-4357-98b5-a871335ee3de-kube-api-access-vfslj\") pod \"service-ca-5fffc97bdf-d49cl\" (UID: \"73b41114-4ff0-4357-98b5-a871335ee3de\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d49cl" Oct 29 22:42:34.594227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594112 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ad00084a-5a9f-489b-9122-2316f617f3f1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:34.594227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594139 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-certificates\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:34.594227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594085 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71be8984-962e-4357-a596-b9d3931e3a75-ovnkube-config\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.594227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594182 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/877326a8-91a3-499f-8b37-4f80dfb00352-trusted-ca\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:34.594227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594196 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hp7m\" (UniqueName: \"kubernetes.io/projected/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656-kube-api-access-7hp7m\") pod \"iptables-alerter-trj6w\" (UID: \"8d2cdca1-7303-4d6c-98ea-5eb6b7d16656\") " pod="openshift-network-operator/iptables-alerter-trj6w" Oct 29 22:42:34.594741 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594255 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cffk\" (UniqueName: \"kubernetes.io/projected/093ceb90-294f-4133-83a6-32548405d530-kube-api-access-5cffk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7klbh\" (UID: \"093ceb90-294f-4133-83a6-32548405d530\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" Oct 29 22:42:34.594741 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594282 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71be8984-962e-4357-a596-b9d3931e3a75-ovnkube-script-lib\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.594741 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594290 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/73b41114-4ff0-4357-98b5-a871335ee3de-signing-key\") pod \"service-ca-5fffc97bdf-d49cl\" (UID: \"73b41114-4ff0-4357-98b5-a871335ee3de\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d49cl" Oct 29 22:42:34.594741 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594339 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-certificates\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:34.594741 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594394 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-bound-sa-token\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:34.594741 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594423 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-etc-openvswitch\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.594741 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594470 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-8ncgh\" (UID: \"96726184-9be2-4685-93d1-415bbb8b551c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:34.594741 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594477 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71be8984-962e-4357-a596-b9d3931e3a75-ovnkube-script-lib\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.594741 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594501 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-socket-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:42:34.594741 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594558 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ad00084a-5a9f-489b-9122-2316f617f3f1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:34.594741 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594586 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-daemon-config\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.594741 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594629 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/884370ff-c54f-4d6b-af78-5c0813bb48eb-installation-pull-secrets\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:34.594741 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594686 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-8ncgh\" (UID: \"96726184-9be2-4685-93d1-415bbb8b551c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:34.594741 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594696 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:34.594741 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594724 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-daemon-config\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.594741 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594731 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71be8984-962e-4357-a596-b9d3931e3a75-env-overrides\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594757 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/877326a8-91a3-499f-8b37-4f80dfb00352-registry-certificates\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594788 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-sysconfig\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594815 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-run-openvswitch\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594838 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/884370ff-c54f-4d6b-af78-5c0813bb48eb-installation-pull-secrets\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594864 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-sys-fs\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594900 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/877326a8-91a3-499f-8b37-4f80dfb00352-installation-pull-secrets\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594966 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-socket-dir-parent\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594991 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-var-lib-kubelet\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.594994 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71be8984-962e-4357-a596-b9d3931e3a75-env-overrides\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595011 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-etc-kubernetes\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595052 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-stats-auth\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595050 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/877326a8-91a3-499f-8b37-4f80dfb00352-registry-certificates\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595108 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85325a00-d2ed-4c7d-b287-0fd5452f7e69-system-cni-dir\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595162 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e91a303-9b80-47f5-a502-fbb6365eaa54-config\") pod \"console-operator-7cd56b8dbc-lm55w\" (UID: \"9e91a303-9b80-47f5-a502-fbb6365eaa54\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595211 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-lib-modules\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595259 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5l7q\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-kube-api-access-t5l7q\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595287 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-tmp\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595263 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-stats-auth\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595327 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/877326a8-91a3-499f-8b37-4f80dfb00352-installation-pull-secrets\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595393 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e91a303-9b80-47f5-a502-fbb6365eaa54-config\") pod \"console-operator-7cd56b8dbc-lm55w\" (UID: \"9e91a303-9b80-47f5-a502-fbb6365eaa54\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595402 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-tmp\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595436 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/266bff54-6027-4822-91ce-be5cbd316ddc-tmp-dir\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595458 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595497 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcb88\" (UniqueName: \"kubernetes.io/projected/4525bd80-124f-4894-9fc0-b8235a0fe689-kube-api-access-gcb88\") pod \"node-ca-rwz4x\" (UID: \"4525bd80-124f-4894-9fc0-b8235a0fe689\") " pod="openshift-image-registry/node-ca-rwz4x" Oct 29 22:42:34.595527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595529 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/266bff54-6027-4822-91ce-be5cbd316ddc-tmp-dir\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595575 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/093ceb90-294f-4133-83a6-32548405d530-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7klbh\" (UID: \"093ceb90-294f-4133-83a6-32548405d530\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595609 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29d1f02-5697-4d94-8a21-0a39e8f473ed-trusted-ca-bundle\") pod \"periodic-gathering-vqgrg-xtvww\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595732 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx7m8\" (UniqueName: \"kubernetes.io/projected/c29d1f02-5697-4d94-8a21-0a39e8f473ed-kube-api-access-cx7m8\") pod \"periodic-gathering-vqgrg-xtvww\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595771 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-log-socket\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595811 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/877326a8-91a3-499f-8b37-4f80dfb00352-image-registry-private-configuration\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595924 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/093ceb90-294f-4133-83a6-32548405d530-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7klbh\" (UID: \"093ceb90-294f-4133-83a6-32548405d530\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595938 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595967 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-kubelet\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595972 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/877326a8-91a3-499f-8b37-4f80dfb00352-image-registry-private-configuration\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595925 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29d1f02-5697-4d94-8a21-0a39e8f473ed-trusted-ca-bundle\") pod \"periodic-gathering-vqgrg-xtvww\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.595990 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71be8984-962e-4357-a596-b9d3931e3a75-ovn-node-metrics-cert\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596070 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-default-certificate\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596091 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-original-pull-secret\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596104 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71be8984-962e-4357-a596-b9d3931e3a75-ovn-node-metrics-cert\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596142 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99577b75-29cd-430b-b5ac-2af3828586d0-service-ca-bundle\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596177 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrls7\" (UniqueName: \"kubernetes.io/projected/85325a00-d2ed-4c7d-b287-0fd5452f7e69-kube-api-access-vrls7\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596244 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-cni-netd\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596257 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-default-certificate\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596277 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596307 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6f91740-e20b-439e-be7d-4b444f4835fa-cert\") pod \"ingress-canary-qbv9d\" (UID: \"d6f91740-e20b-439e-be7d-4b444f4835fa\") " pod="openshift-ingress-canary/ingress-canary-qbv9d" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596344 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a109715c-c29a-48d1-a573-3e7800f48259-config\") pod \"service-ca-operator-544d749c8c-j74gf\" (UID: \"a109715c-c29a-48d1-a573-3e7800f48259\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596375 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e91a303-9b80-47f5-a502-fbb6365eaa54-serving-cert\") pod \"console-operator-7cd56b8dbc-lm55w\" (UID: \"9e91a303-9b80-47f5-a502-fbb6365eaa54\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596402 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-serving-cert\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596427 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/266bff54-6027-4822-91ce-be5cbd316ddc-config-volume\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596454 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596480 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/575ee3c4-7c77-4b31-9961-eebc60254d19-nginx-conf\") pod \"networking-console-plugin-86c478bb88-rwfmh\" (UID: \"575ee3c4-7c77-4b31-9961-eebc60254d19\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rwfmh" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596491 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6f91740-e20b-439e-be7d-4b444f4835fa-cert\") pod \"ingress-canary-qbv9d\" (UID: \"d6f91740-e20b-439e-be7d-4b444f4835fa\") " pod="openshift-ingress-canary/ingress-canary-qbv9d" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596541 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-run-k8s-cni-cncf-io\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596547 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a109715c-c29a-48d1-a573-3e7800f48259-config\") pod \"service-ca-operator-544d749c8c-j74gf\" (UID: \"a109715c-c29a-48d1-a573-3e7800f48259\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596564 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-metrics-certs\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596604 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4525bd80-124f-4894-9fc0-b8235a0fe689-host\") pod \"node-ca-rwz4x\" (UID: \"4525bd80-124f-4894-9fc0-b8235a0fe689\") " pod="openshift-image-registry/node-ca-rwz4x" Oct 29 22:42:34.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596625 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z9zh\" (UniqueName: \"kubernetes.io/projected/af926a18-1147-4786-8bd9-8a6727abb6fa-kube-api-access-2z9zh\") pod \"node-resolver-5czm4\" (UID: \"af926a18-1147-4786-8bd9-8a6727abb6fa\") " pod="openshift-dns/node-resolver-5czm4" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596657 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-kubelet-config\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596682 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e91a303-9b80-47f5-a502-fbb6365eaa54-trusted-ca\") pod \"console-operator-7cd56b8dbc-lm55w\" (UID: \"9e91a303-9b80-47f5-a502-fbb6365eaa54\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596693 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/266bff54-6027-4822-91ce-be5cbd316ddc-config-volume\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596697 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-serving-cert\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596708 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs\") pod \"network-metrics-daemon-mxg8q\" (UID: \"286133c3-abe5-4954-83a1-8665c1785b92\") " pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596773 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcjhj\" (UniqueName: \"kubernetes.io/projected/6a4e4c73-0516-464c-9b6e-1c4adf93c03b-kube-api-access-dcjhj\") pod \"kas-connection-checker-685895c7c9-7hz4v\" (UID: \"6a4e4c73-0516-464c-9b6e-1c4adf93c03b\") " pod="kube-system/kas-connection-checker-685895c7c9-7hz4v" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596819 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e91a303-9b80-47f5-a502-fbb6365eaa54-serving-cert\") pod \"console-operator-7cd56b8dbc-lm55w\" (UID: \"9e91a303-9b80-47f5-a502-fbb6365eaa54\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596833 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/575ee3c4-7c77-4b31-9961-eebc60254d19-nginx-conf\") pod \"networking-console-plugin-86c478bb88-rwfmh\" (UID: \"575ee3c4-7c77-4b31-9961-eebc60254d19\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rwfmh" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596846 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-systemd-units\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596897 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcfvz\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-kube-api-access-hcfvz\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596934 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-cni-dir\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.596961 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597001 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e91a303-9b80-47f5-a502-fbb6365eaa54-trusted-ca\") pod \"console-operator-7cd56b8dbc-lm55w\" (UID: \"9e91a303-9b80-47f5-a502-fbb6365eaa54\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597003 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/884370ff-c54f-4d6b-af78-5c0813bb48eb-ca-trust-extracted\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597049 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5d85\" (UniqueName: \"kubernetes.io/projected/9483bf09-4676-47db-a279-84aabe9d17ed-kube-api-access-w5d85\") pod \"migrator-66df5d496b-zl99j\" (UID: \"9483bf09-4676-47db-a279-84aabe9d17ed\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zl99j" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597077 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/884370ff-c54f-4d6b-af78-5c0813bb48eb-ca-trust-extracted\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597202 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-run-ovn-kubernetes\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597236 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597261 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvrrh\" (UniqueName: \"kubernetes.io/projected/286133c3-abe5-4954-83a1-8665c1785b92-kube-api-access-qvrrh\") pod \"network-metrics-daemon-mxg8q\" (UID: \"286133c3-abe5-4954-83a1-8665c1785b92\") " pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597275 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af926a18-1147-4786-8bd9-8a6727abb6fa-hosts-file\") pod \"node-resolver-5czm4\" (UID: \"af926a18-1147-4786-8bd9-8a6727abb6fa\") " pod="openshift-dns/node-resolver-5czm4" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597300 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhmrs\" (UniqueName: \"kubernetes.io/projected/71be8984-962e-4357-a596-b9d3931e3a75-kube-api-access-rhmrs\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597322 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk8rz\" (UniqueName: \"kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz\") pod \"network-check-target-j8fft\" (UID: \"463bc340-b1dc-41e6-9203-fcd2672f8a17\") " pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597340 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597354 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656-host-slash\") pod \"iptables-alerter-trj6w\" (UID: \"8d2cdca1-7303-4d6c-98ea-5eb6b7d16656\") " pod="openshift-network-operator/iptables-alerter-trj6w" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597387 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597512 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-registration-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597543 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-dbus\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597616 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kpn4\" (UniqueName: \"kubernetes.io/projected/ad00084a-5a9f-489b-9122-2316f617f3f1-kube-api-access-2kpn4\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597681 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97fa3566-0c3f-4768-a437-3a89816303e0-tmp\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597725 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-var-lib-openvswitch\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597750 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-hostroot\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597775 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85325a00-d2ed-4c7d-b287-0fd5452f7e69-cni-binary-copy\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597788 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97fa3566-0c3f-4768-a437-3a89816303e0-tmp\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597801 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-kubernetes\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597829 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4525bd80-124f-4894-9fc0-b8235a0fe689-serviceca\") pod \"node-ca-rwz4x\" (UID: \"4525bd80-124f-4894-9fc0-b8235a0fe689\") " pod="openshift-image-registry/node-ca-rwz4x" Oct 29 22:42:34.597905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597899 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a109715c-c29a-48d1-a573-3e7800f48259-serving-cert\") pod \"service-ca-operator-544d749c8c-j74gf\" (UID: \"a109715c-c29a-48d1-a573-3e7800f48259\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.597973 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85325a00-d2ed-4c7d-b287-0fd5452f7e69-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598000 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82r6j\" (UniqueName: \"kubernetes.io/projected/266bff54-6027-4822-91ce-be5cbd316ddc-kube-api-access-82r6j\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598023 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4525bd80-124f-4894-9fc0-b8235a0fe689-serviceca\") pod \"node-ca-rwz4x\" (UID: \"4525bd80-124f-4894-9fc0-b8235a0fe689\") " pod="openshift-image-registry/node-ca-rwz4x" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598026 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c29d1f02-5697-4d94-8a21-0a39e8f473ed-archives-path\") pod \"periodic-gathering-vqgrg-xtvww\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598063 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-device-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598084 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85325a00-d2ed-4c7d-b287-0fd5452f7e69-cni-binary-copy\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598118 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c29d1f02-5697-4d94-8a21-0a39e8f473ed-archives-path\") pod \"periodic-gathering-vqgrg-xtvww\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598139 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h744b\" (UniqueName: \"kubernetes.io/projected/99577b75-29cd-430b-b5ac-2af3828586d0-kube-api-access-h744b\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598199 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/884370ff-c54f-4d6b-af78-5c0813bb48eb-trusted-ca\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598224 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-host\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598249 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-run-ovn\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598271 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85325a00-d2ed-4c7d-b287-0fd5452f7e69-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598274 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/73b41114-4ff0-4357-98b5-a871335ee3de-signing-cabundle\") pod \"service-ca-5fffc97bdf-d49cl\" (UID: \"73b41114-4ff0-4357-98b5-a871335ee3de\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d49cl" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598315 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9-agent-certs\") pod \"konnectivity-agent-sc55n\" (UID: \"2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9\") " pod="kube-system/konnectivity-agent-sc55n" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598322 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a109715c-c29a-48d1-a573-3e7800f48259-serving-cert\") pod \"service-ca-operator-544d749c8c-j74gf\" (UID: \"a109715c-c29a-48d1-a573-3e7800f48259\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598349 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9-konnectivity-ca\") pod \"konnectivity-agent-sc55n\" (UID: \"2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9\") " pod="kube-system/konnectivity-agent-sc55n" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598372 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-bound-sa-token\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598408 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-var-lib-kubelet\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598434 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29d1f02-5697-4d94-8a21-0a39e8f473ed-service-ca-bundle\") pod \"periodic-gathering-vqgrg-xtvww\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598445 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/73b41114-4ff0-4357-98b5-a871335ee3de-signing-cabundle\") pod \"service-ca-5fffc97bdf-d49cl\" (UID: \"73b41114-4ff0-4357-98b5-a871335ee3de\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d49cl" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598467 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-run-netns\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598499 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-crio-socket\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598535 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz8wx\" (UniqueName: \"kubernetes.io/projected/68daf26a-81a0-4bd5-a188-9b39a1f460a7-kube-api-access-rz8wx\") pod \"kas-connection-checker-685895c7c9-k8xnq\" (UID: \"68daf26a-81a0-4bd5-a188-9b39a1f460a7\") " pod="kube-system/kas-connection-checker-685895c7c9-k8xnq" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598548 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vllbq\"" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598583 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/093ceb90-294f-4133-83a6-32548405d530-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7klbh\" (UID: \"093ceb90-294f-4133-83a6-32548405d530\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598589 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29d1f02-5697-4d94-8a21-0a39e8f473ed-service-ca-bundle\") pod \"periodic-gathering-vqgrg-xtvww\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598612 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-slash\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598617 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9-konnectivity-ca\") pod \"konnectivity-agent-sc55n\" (UID: \"2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9\") " pod="kube-system/konnectivity-agent-sc55n" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598641 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598648 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/884370ff-c54f-4d6b-af78-5c0813bb48eb-trusted-ca\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598659 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9-agent-certs\") pod \"konnectivity-agent-sc55n\" (UID: \"2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9\") " pod="kube-system/konnectivity-agent-sc55n" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598705 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx2rx\" (UniqueName: \"kubernetes.io/projected/d6f91740-e20b-439e-be7d-4b444f4835fa-kube-api-access-rx2rx\") pod \"ingress-canary-qbv9d\" (UID: \"d6f91740-e20b-439e-be7d-4b444f4835fa\") " pod="openshift-ingress-canary/ingress-canary-qbv9d" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598741 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598765 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/093ceb90-294f-4133-83a6-32548405d530-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7klbh\" (UID: \"093ceb90-294f-4133-83a6-32548405d530\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598775 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/97fa3566-0c3f-4768-a437-3a89816303e0-etc-tuned\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598821 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-data-volume\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598836 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/97fa3566-0c3f-4768-a437-3a89816303e0-etc-tuned\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598864 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg6m4\" (UniqueName: \"kubernetes.io/projected/96726184-9be2-4685-93d1-415bbb8b551c-kube-api-access-cg6m4\") pod \"cluster-samples-operator-6ddb6c8b94-8ncgh\" (UID: \"96726184-9be2-4685-93d1-415bbb8b551c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598907 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-data-volume\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598946 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgc6n\" (UniqueName: \"kubernetes.io/projected/a1a232f4-f67c-404f-a1f7-af0c3cc30180-kube-api-access-kgc6n\") pod \"kas-connection-checker-685895c7c9-fh7rx\" (UID: \"a1a232f4-f67c-404f-a1f7-af0c3cc30180\") " pod="kube-system/kas-connection-checker-685895c7c9-fh7rx" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598963 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85325a00-d2ed-4c7d-b287-0fd5452f7e69-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.598992 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-modprobe-d\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.599017 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-sysctl-d\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.599142 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plrh7\" (UniqueName: \"kubernetes.io/projected/97fa3566-0c3f-4768-a437-3a89816303e0-kube-api-access-plrh7\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.599232 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-run-multus-certs\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.599245 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85325a00-d2ed-4c7d-b287-0fd5452f7e69-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.599298 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-service-ca-bundle\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:34.599303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.599318 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-sysctl-conf\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.600748 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.599337 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9l7w\" (UniqueName: \"kubernetes.io/projected/85ab246f-f679-40c5-a127-c0fd5a4a621e-kube-api-access-g9l7w\") pod \"network-check-source-6f78d59f99-kw5c9\" (UID: \"85ab246f-f679-40c5-a127-c0fd5a4a621e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kw5c9" Oct 29 22:42:34.600748 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.599355 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdmbj\" (UniqueName: \"kubernetes.io/projected/4590699c-6eb3-4546-835f-924180048d8a-kube-api-access-jdmbj\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:42:34.600748 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.599443 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-service-ca-bundle\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:34.600748 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.599462 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-var-lib-cni-multus\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.600748 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.599492 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:34.600748 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.599515 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-run\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.600748 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.599581 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqxqz\" (UniqueName: \"kubernetes.io/projected/a109715c-c29a-48d1-a573-3e7800f48259-kube-api-access-zqxqz\") pod \"service-ca-operator-544d749c8c-j74gf\" (UID: \"a109715c-c29a-48d1-a573-3e7800f48259\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" Oct 29 22:42:34.600748 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.599612 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/575ee3c4-7c77-4b31-9961-eebc60254d19-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rwfmh\" (UID: \"575ee3c4-7c77-4b31-9961-eebc60254d19\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rwfmh" Oct 29 22:42:34.600748 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.599640 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m76zp\" (UniqueName: \"kubernetes.io/projected/8f6c01b9-e044-413c-b533-09033d04f29e-kube-api-access-m76zp\") pod \"volume-data-source-validator-58c9f5f46d-868dt\" (UID: \"8f6c01b9-e044-413c-b533-09033d04f29e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-868dt" Oct 29 22:42:34.600748 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.599671 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4z7h\" (UniqueName: \"kubernetes.io/projected/ce5dd070-fce7-44da-aaeb-017109fc98c9-kube-api-access-h4z7h\") pod \"migrator-66df5d496b-n56zr\" (UID: \"ce5dd070-fce7-44da-aaeb-017109fc98c9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n56zr" Oct 29 22:42:34.600748 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.599694 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-run-netns\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.600748 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.599722 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/884370ff-c54f-4d6b-af78-5c0813bb48eb-image-registry-private-configuration\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:34.600748 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.599750 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6nxm\" (UniqueName: \"kubernetes.io/projected/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-kube-api-access-f6nxm\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:34.600748 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.599774 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-run-systemd\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.600748 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.600031 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/884370ff-c54f-4d6b-af78-5c0813bb48eb-image-registry-private-configuration\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:34.616351 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.616331 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvm7r\" (UniqueName: \"kubernetes.io/projected/9e91a303-9b80-47f5-a502-fbb6365eaa54-kube-api-access-zvm7r\") pod \"console-operator-7cd56b8dbc-lm55w\" (UID: \"9e91a303-9b80-47f5-a502-fbb6365eaa54\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:34.616611 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.616591 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrhns\" (UniqueName: \"kubernetes.io/projected/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-api-access-jrhns\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:34.621659 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.621633 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85sqm\" (UniqueName: \"kubernetes.io/projected/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-kube-api-access-85sqm\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.621817 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.621796 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hp7m\" (UniqueName: \"kubernetes.io/projected/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656-kube-api-access-7hp7m\") pod \"iptables-alerter-trj6w\" (UID: \"8d2cdca1-7303-4d6c-98ea-5eb6b7d16656\") " pod="openshift-network-operator/iptables-alerter-trj6w" Oct 29 22:42:34.621865 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.621806 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cffk\" (UniqueName: \"kubernetes.io/projected/093ceb90-294f-4133-83a6-32548405d530-kube-api-access-5cffk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7klbh\" (UID: \"093ceb90-294f-4133-83a6-32548405d530\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" Oct 29 22:42:34.621966 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.621943 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-bound-sa-token\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:34.622106 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.622091 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfslj\" (UniqueName: \"kubernetes.io/projected/73b41114-4ff0-4357-98b5-a871335ee3de-kube-api-access-vfslj\") pod \"service-ca-5fffc97bdf-d49cl\" (UID: \"73b41114-4ff0-4357-98b5-a871335ee3de\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d49cl" Oct 29 22:42:34.622627 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.622611 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvrrh\" (UniqueName: \"kubernetes.io/projected/286133c3-abe5-4954-83a1-8665c1785b92-kube-api-access-qvrrh\") pod \"network-metrics-daemon-mxg8q\" (UID: \"286133c3-abe5-4954-83a1-8665c1785b92\") " pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:34.627116 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.627094 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx7m8\" (UniqueName: \"kubernetes.io/projected/c29d1f02-5697-4d94-8a21-0a39e8f473ed-kube-api-access-cx7m8\") pod \"periodic-gathering-vqgrg-xtvww\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:42:34.627911 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.627888 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z9zh\" (UniqueName: \"kubernetes.io/projected/af926a18-1147-4786-8bd9-8a6727abb6fa-kube-api-access-2z9zh\") pod \"node-resolver-5czm4\" (UID: \"af926a18-1147-4786-8bd9-8a6727abb6fa\") " pod="openshift-dns/node-resolver-5czm4" Oct 29 22:42:34.627911 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.627899 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5l7q\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-kube-api-access-t5l7q\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:34.628605 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.628585 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcjhj\" (UniqueName: \"kubernetes.io/projected/6a4e4c73-0516-464c-9b6e-1c4adf93c03b-kube-api-access-dcjhj\") pod \"kas-connection-checker-685895c7c9-7hz4v\" (UID: \"6a4e4c73-0516-464c-9b6e-1c4adf93c03b\") " pod="kube-system/kas-connection-checker-685895c7c9-7hz4v" Oct 29 22:42:34.628754 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.628735 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5d85\" (UniqueName: \"kubernetes.io/projected/9483bf09-4676-47db-a279-84aabe9d17ed-kube-api-access-w5d85\") pod \"migrator-66df5d496b-zl99j\" (UID: \"9483bf09-4676-47db-a279-84aabe9d17ed\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zl99j" Oct 29 22:42:34.628813 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.628774 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk8rz\" (UniqueName: \"kubernetes.io/projected/463bc340-b1dc-41e6-9203-fcd2672f8a17-kube-api-access-jk8rz\") pod \"network-check-target-j8fft\" (UID: \"463bc340-b1dc-41e6-9203-fcd2672f8a17\") " pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:34.634309 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.633614 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82r6j\" (UniqueName: \"kubernetes.io/projected/266bff54-6027-4822-91ce-be5cbd316ddc-kube-api-access-82r6j\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:34.634309 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.633724 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdmbj\" (UniqueName: \"kubernetes.io/projected/4590699c-6eb3-4546-835f-924180048d8a-kube-api-access-jdmbj\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:42:34.634309 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.634031 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqxqz\" (UniqueName: \"kubernetes.io/projected/a109715c-c29a-48d1-a573-3e7800f48259-kube-api-access-zqxqz\") pod \"service-ca-operator-544d749c8c-j74gf\" (UID: \"a109715c-c29a-48d1-a573-3e7800f48259\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" Oct 29 22:42:34.634309 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.634273 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcb88\" (UniqueName: \"kubernetes.io/projected/4525bd80-124f-4894-9fc0-b8235a0fe689-kube-api-access-gcb88\") pod \"node-ca-rwz4x\" (UID: \"4525bd80-124f-4894-9fc0-b8235a0fe689\") " pod="openshift-image-registry/node-ca-rwz4x" Oct 29 22:42:34.640271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.639747 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrls7\" (UniqueName: \"kubernetes.io/projected/85325a00-d2ed-4c7d-b287-0fd5452f7e69-kube-api-access-vrls7\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:42:34.640271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.639780 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgc6n\" (UniqueName: \"kubernetes.io/projected/a1a232f4-f67c-404f-a1f7-af0c3cc30180-kube-api-access-kgc6n\") pod \"kas-connection-checker-685895c7c9-fh7rx\" (UID: \"a1a232f4-f67c-404f-a1f7-af0c3cc30180\") " pod="kube-system/kas-connection-checker-685895c7c9-fh7rx" Oct 29 22:42:34.640271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.639867 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx2rx\" (UniqueName: \"kubernetes.io/projected/d6f91740-e20b-439e-be7d-4b444f4835fa-kube-api-access-rx2rx\") pod \"ingress-canary-qbv9d\" (UID: \"d6f91740-e20b-439e-be7d-4b444f4835fa\") " pod="openshift-ingress-canary/ingress-canary-qbv9d" Oct 29 22:42:34.640271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.639984 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcfvz\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-kube-api-access-hcfvz\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:34.640271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.640073 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kpn4\" (UniqueName: \"kubernetes.io/projected/ad00084a-5a9f-489b-9122-2316f617f3f1-kube-api-access-2kpn4\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:34.640271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.640121 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h744b\" (UniqueName: \"kubernetes.io/projected/99577b75-29cd-430b-b5ac-2af3828586d0-kube-api-access-h744b\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:34.640271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.640176 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhmrs\" (UniqueName: \"kubernetes.io/projected/71be8984-962e-4357-a596-b9d3931e3a75-kube-api-access-rhmrs\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.640271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.640220 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-bound-sa-token\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:34.640271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.640222 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg6m4\" (UniqueName: \"kubernetes.io/projected/96726184-9be2-4685-93d1-415bbb8b551c-kube-api-access-cg6m4\") pod \"cluster-samples-operator-6ddb6c8b94-8ncgh\" (UID: \"96726184-9be2-4685-93d1-415bbb8b551c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:34.640747 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.640708 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plrh7\" (UniqueName: \"kubernetes.io/projected/97fa3566-0c3f-4768-a437-3a89816303e0-kube-api-access-plrh7\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.643778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.641037 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4z7h\" (UniqueName: \"kubernetes.io/projected/ce5dd070-fce7-44da-aaeb-017109fc98c9-kube-api-access-h4z7h\") pod \"migrator-66df5d496b-n56zr\" (UID: \"ce5dd070-fce7-44da-aaeb-017109fc98c9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n56zr" Oct 29 22:42:34.643778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.641092 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz8wx\" (UniqueName: \"kubernetes.io/projected/68daf26a-81a0-4bd5-a188-9b39a1f460a7-kube-api-access-rz8wx\") pod \"kas-connection-checker-685895c7c9-k8xnq\" (UID: \"68daf26a-81a0-4bd5-a188-9b39a1f460a7\") " pod="kube-system/kas-connection-checker-685895c7c9-k8xnq" Oct 29 22:42:34.643778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.641471 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6nxm\" (UniqueName: \"kubernetes.io/projected/aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8-kube-api-access-f6nxm\") pod \"insights-operator-7c4986d449-9qhv9\" (UID: \"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8\") " pod="openshift-insights/insights-operator-7c4986d449-9qhv9" Oct 29 22:42:34.643778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.641564 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9l7w\" (UniqueName: \"kubernetes.io/projected/85ab246f-f679-40c5-a127-c0fd5a4a621e-kube-api-access-g9l7w\") pod \"network-check-source-6f78d59f99-kw5c9\" (UID: \"85ab246f-f679-40c5-a127-c0fd5a4a621e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kw5c9" Oct 29 22:42:34.643778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.641620 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m76zp\" (UniqueName: \"kubernetes.io/projected/8f6c01b9-e044-413c-b533-09033d04f29e-kube-api-access-m76zp\") pod \"volume-data-source-validator-58c9f5f46d-868dt\" (UID: \"8f6c01b9-e044-413c-b533-09033d04f29e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-868dt" Oct 29 22:42:34.700496 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700416 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-socket-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:42:34.700496 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700453 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-sysconfig\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.700496 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700469 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-run-openvswitch\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.700496 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700485 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-sys-fs\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:42:34.700775 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700504 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-socket-dir-parent\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.700775 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700527 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-var-lib-kubelet\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.700775 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700560 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-run-openvswitch\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.700775 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700570 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-etc-kubernetes\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.700775 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700570 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-socket-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:42:34.700775 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700595 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85325a00-d2ed-4c7d-b287-0fd5452f7e69-system-cni-dir\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:42:34.700775 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700599 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-sys-fs\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:42:34.700775 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700616 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-lib-modules\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.700775 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700600 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-var-lib-kubelet\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.700775 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700626 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-etc-kubernetes\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.700775 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700667 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85325a00-d2ed-4c7d-b287-0fd5452f7e69-system-cni-dir\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:42:34.700775 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700717 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-lib-modules\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.700775 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700725 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-log-socket\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.700775 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700751 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-log-socket\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.700775 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700755 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-kubelet\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.700775 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700775 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-kubelet\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.700775 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700781 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99577b75-29cd-430b-b5ac-2af3828586d0-service-ca-bundle\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700806 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-cni-netd\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700831 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700859 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700878 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-cni-netd\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700884 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-run-k8s-cni-cncf-io\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700916 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-run-k8s-cni-cncf-io\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700919 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-metrics-certs\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700949 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4525bd80-124f-4894-9fc0-b8235a0fe689-host\") pod \"node-ca-rwz4x\" (UID: \"4525bd80-124f-4894-9fc0-b8235a0fe689\") " pod="openshift-image-registry/node-ca-rwz4x" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700948 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.700976 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-kubelet-config\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701005 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-socket-dir-parent\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701047 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4525bd80-124f-4894-9fc0-b8235a0fe689-host\") pod \"node-ca-rwz4x\" (UID: \"4525bd80-124f-4894-9fc0-b8235a0fe689\") " pod="openshift-image-registry/node-ca-rwz4x" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701051 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs\") pod \"network-metrics-daemon-mxg8q\" (UID: \"286133c3-abe5-4954-83a1-8665c1785b92\") " pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701093 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-systemd-units\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:34.701115 5057 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701109 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-sysconfig\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701173 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-cni-dir\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701119 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-cni-dir\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701204 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-kubelet-config\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:34.701218 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls podName:266bff54-6027-4822-91ce-be5cbd316ddc nodeName:}" failed. No retries permitted until 2026-10-29 22:42:35.201173353 +0000 UTC m=+1.852671866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls") pod "dns-default-gj6vs" (UID: "266bff54-6027-4822-91ce-be5cbd316ddc") : secret "dns-default-metrics-tls" not found Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701253 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-systemd-units\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701305 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701336 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-run-ovn-kubernetes\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701361 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701386 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af926a18-1147-4786-8bd9-8a6727abb6fa-hosts-file\") pod \"node-resolver-5czm4\" (UID: \"af926a18-1147-4786-8bd9-8a6727abb6fa\") " pod="openshift-dns/node-resolver-5czm4" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701412 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:34.701485 5057 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 22:42:34.701520 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:34.701497 5057 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701427 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-run-ovn-kubernetes\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701526 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99577b75-29cd-430b-b5ac-2af3828586d0-service-ca-bundle\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701469 5057 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:34.701504 5057 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:34.701545 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm podName:d518d30f-c46b-49ca-b74c-9f4cf0ca71e9 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:35.201530075 +0000 UTC m=+1.853028612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6rqj6" (UID: "d518d30f-c46b-49ca-b74c-9f4cf0ca71e9") : configmap "kube-rbac-proxy" not found Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701550 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af926a18-1147-4786-8bd9-8a6727abb6fa-hosts-file\") pod \"node-resolver-5czm4\" (UID: \"af926a18-1147-4786-8bd9-8a6727abb6fa\") " pod="openshift-dns/node-resolver-5czm4" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701708 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656-host-slash\") pod \"iptables-alerter-trj6w\" (UID: \"8d2cdca1-7303-4d6c-98ea-5eb6b7d16656\") " pod="openshift-network-operator/iptables-alerter-trj6w" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701739 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701748 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d2cdca1-7303-4d6c-98ea-5eb6b7d16656-host-slash\") pod \"iptables-alerter-trj6w\" (UID: \"8d2cdca1-7303-4d6c-98ea-5eb6b7d16656\") " pod="openshift-network-operator/iptables-alerter-trj6w" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:34.701776 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls podName:ad00084a-5a9f-489b-9122-2316f617f3f1 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:35.201754821 +0000 UTC m=+1.853253335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dzx6b" (UID: "ad00084a-5a9f-489b-9122-2316f617f3f1") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701788 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:34.701813 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls podName:d518d30f-c46b-49ca-b74c-9f4cf0ca71e9 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:35.201803367 +0000 UTC m=+1.853301881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6rqj6" (UID: "d518d30f-c46b-49ca-b74c-9f4cf0ca71e9") : secret "insights-runtime-extractor-tls" not found Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701833 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-registration-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701860 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-dbus\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701888 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-var-lib-openvswitch\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701909 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-hostroot\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701932 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-kubernetes\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701961 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-device-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.701984 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-host\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702006 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-run-ovn\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702037 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-var-lib-kubelet\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702061 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-run-netns\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702086 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-crio-socket\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702110 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-slash\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702133 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702158 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702210 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-modprobe-d\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702217 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-device-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702235 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-sysctl-d\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702260 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-run-multus-certs\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702284 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-sysctl-conf\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702297 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-registration-dir\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702309 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-var-lib-cni-multus\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702335 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702356 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-run\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702373 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41ea529e-ab5d-419c-87bf-fe0bd1a6460f-dbus\") pod \"global-pull-secret-syncer-b2cf2\" (UID: \"41ea529e-ab5d-419c-87bf-fe0bd1a6460f\") " pod="kube-system/global-pull-secret-syncer-b2cf2" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702380 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-hostroot\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702381 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/575ee3c4-7c77-4b31-9961-eebc60254d19-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rwfmh\" (UID: \"575ee3c4-7c77-4b31-9961-eebc60254d19\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rwfmh" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702445 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-kubernetes\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702618 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-sysctl-conf\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:34.702643 5057 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:34.702657 5057 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf56cb474-gx24x: secret "image-registry-tls" not found Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:34.702720 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls podName:877326a8-91a3-499f-8b37-4f80dfb00352 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:35.202705742 +0000 UTC m=+1.854204281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls") pod "image-registry-5cf56cb474-gx24x" (UID: "877326a8-91a3-499f-8b37-4f80dfb00352") : secret "image-registry-tls" not found Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702742 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-modprobe-d\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702785 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-host\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702844 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-run-ovn\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.702868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702848 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-sysctl-d\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702908 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-run-multus-certs\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702898 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-var-lib-kubelet\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702960 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-slash\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.702983 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-crio-socket\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703025 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4590699c-6eb3-4546-835f-924180048d8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-pb5dp\" (UID: \"4590699c-6eb3-4546-835f-924180048d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pb5dp" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703026 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-run-netns\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703095 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-var-lib-cni-multus\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703159 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-run\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:34.703175 5057 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:34.703212 5057 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f4dff4c9-87jff: secret "image-registry-tls" not found Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703237 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-var-lib-openvswitch\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:34.703267 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls podName:884370ff-c54f-4d6b-af78-5c0813bb48eb nodeName:}" failed. No retries permitted until 2026-10-29 22:42:35.203252265 +0000 UTC m=+1.854750775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls") pod "image-registry-78f4dff4c9-87jff" (UID: "884370ff-c54f-4d6b-af78-5c0813bb48eb") : secret "image-registry-tls" not found Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703324 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-run-netns\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703361 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-run-systemd\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703385 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-var-lib-cni-bin\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703407 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85325a00-d2ed-4c7d-b287-0fd5452f7e69-cnibin\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703419 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-run-netns\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703433 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8ncgh\" (UID: \"96726184-9be2-4685-93d1-415bbb8b551c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703492 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85325a00-d2ed-4c7d-b287-0fd5452f7e69-cnibin\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703542 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-host-var-lib-cni-bin\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703573 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-conf-dir\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703663 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-multus-conf-dir\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703722 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-node-log\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703751 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85325a00-d2ed-4c7d-b287-0fd5452f7e69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703799 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-sys\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703827 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-cni-bin\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703874 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85325a00-d2ed-4c7d-b287-0fd5452f7e69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vvp9h\" (UID: \"85325a00-d2ed-4c7d-b287-0fd5452f7e69\") " pod="openshift-multus/multus-additional-cni-plugins-vvp9h" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703875 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-cnibin\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703915 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-systemd\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703920 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-cnibin\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703973 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-node-log\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703987 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-system-cni-dir\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.704015 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-etc-systemd\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.704052 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-host-cni-bin\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.704055 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97fa3566-0c3f-4768-a437-3a89816303e0-sys\") pod \"tuned-flzmd\" (UID: \"97fa3566-0c3f-4768-a437-3a89816303e0\") " pod="openshift-cluster-node-tuning-operator/tuned-flzmd" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.703940 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2863700d-26b0-4e9d-bd6a-ce8939cf3c9a-system-cni-dir\") pod \"multus-qqk5c\" (UID: \"2863700d-26b0-4e9d-bd6a-ce8939cf3c9a\") " pod="openshift-multus/multus-qqk5c" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.704095 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-etc-openvswitch\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.704116 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-run-systemd\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.704234 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71be8984-962e-4357-a596-b9d3931e3a75-etc-openvswitch\") pod \"ovnkube-node-99ngw\" (UID: \"71be8984-962e-4357-a596-b9d3931e3a75\") " pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.704633 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/286133c3-abe5-4954-83a1-8665c1785b92-metrics-certs\") pod \"network-metrics-daemon-mxg8q\" (UID: \"286133c3-abe5-4954-83a1-8665c1785b92\") " pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:34.704934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.704903 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99577b75-29cd-430b-b5ac-2af3828586d0-metrics-certs\") pod \"router-default-7945df5b5-pcgp6\" (UID: \"99577b75-29cd-430b-b5ac-2af3828586d0\") " pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:34.706264 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.705482 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/575ee3c4-7c77-4b31-9961-eebc60254d19-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rwfmh\" (UID: \"575ee3c4-7c77-4b31-9961-eebc60254d19\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rwfmh" Oct 29 22:42:34.706464 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.706445 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96726184-9be2-4685-93d1-415bbb8b551c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8ncgh\" (UID: \"96726184-9be2-4685-93d1-415bbb8b551c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:34.782496 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.782465 5057 scope.go:122] "RemoveContainer" containerID="f0fcd8c76c7857be846eac042de3c1bf7b719752c31262e9fa639c35cfdc6aac" Oct 29 22:42:34.784685 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.784663 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:34.784824 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.784807 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" Oct 29 22:42:34.785894 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.785881 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-rwfmh" Oct 29 22:42:34.797227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.797182 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mxg8q" Oct 29 22:42:34.986916 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:34.986852 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qbv9d_d6f91740-e20b-439e-be7d-4b444f4835fa/serve-healthcheck-canary/0.log" Oct 29 22:42:35.092424 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.092397 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:35.124690 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.124654 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7945df5b5-pcgp6"] Oct 29 22:42:35.125811 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:35.125783 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99577b75_29cd_430b_b5ac_2af3828586d0.slice/crio-874e2dc343e4c79c8132d99affde5881940223e66ee73b998a5914c1e76ccc66 WatchSource:0}: Error finding container 874e2dc343e4c79c8132d99affde5881940223e66ee73b998a5914c1e76ccc66: Status 404 returned error can't find the container with id 874e2dc343e4c79c8132d99affde5881940223e66ee73b998a5914c1e76ccc66 Oct 29 22:42:35.160699 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.160675 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh"] Oct 29 22:42:35.185601 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.185579 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mxg8q"] Oct 29 22:42:35.189464 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:35.188229 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod286133c3_abe5_4954_83a1_8665c1785b92.slice/crio-ecb515a19180a745bdd6c1b4168a3b7d02c0e6de55911e4f43400ce1ed2822c1 WatchSource:0}: Error finding container ecb515a19180a745bdd6c1b4168a3b7d02c0e6de55911e4f43400ce1ed2822c1: Status 404 returned error can't find the container with id ecb515a19180a745bdd6c1b4168a3b7d02c0e6de55911e4f43400ce1ed2822c1 Oct 29 22:42:35.190875 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.190854 5057 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:42:35.199365 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.199339 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-n56zr_ce5dd070-fce7-44da-aaeb-017109fc98c9/migrator/0.log" Oct 29 22:42:35.208789 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.208762 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:35.208885 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.208808 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:35.208885 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.208842 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:35.208982 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.208898 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:35.208982 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:35.208930 5057 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:42:35.208982 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:35.208973 5057 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:42:35.209061 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:35.209020 5057 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:42:35.209061 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:35.209037 5057 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f4dff4c9-87jff: secret "image-registry-tls" not found Oct 29 22:42:35.209061 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.208936 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:35.209061 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:35.209049 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls podName:ad00084a-5a9f-489b-9122-2316f617f3f1 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:36.209013232 +0000 UTC m=+2.860511743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dzx6b" (UID: "ad00084a-5a9f-489b-9122-2316f617f3f1") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:42:35.209250 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:35.209081 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls podName:d518d30f-c46b-49ca-b74c-9f4cf0ca71e9 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:36.209064427 +0000 UTC m=+2.860562956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6rqj6" (UID: "d518d30f-c46b-49ca-b74c-9f4cf0ca71e9") : secret "insights-runtime-extractor-tls" not found Oct 29 22:42:35.209250 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:35.209106 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls podName:884370ff-c54f-4d6b-af78-5c0813bb48eb nodeName:}" failed. No retries permitted until 2026-10-29 22:42:36.209094008 +0000 UTC m=+2.860592516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls") pod "image-registry-78f4dff4c9-87jff" (UID: "884370ff-c54f-4d6b-af78-5c0813bb48eb") : secret "image-registry-tls" not found Oct 29 22:42:35.209250 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:35.209138 5057 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 22:42:35.209250 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:35.209163 5057 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:42:35.209250 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:35.209172 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm podName:d518d30f-c46b-49ca-b74c-9f4cf0ca71e9 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:36.209161627 +0000 UTC m=+2.860660137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6rqj6" (UID: "d518d30f-c46b-49ca-b74c-9f4cf0ca71e9") : configmap "kube-rbac-proxy" not found Oct 29 22:42:35.209250 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:35.209172 5057 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf56cb474-gx24x: secret "image-registry-tls" not found Oct 29 22:42:35.209553 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.209256 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:35.209553 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:35.209267 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls podName:877326a8-91a3-499f-8b37-4f80dfb00352 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:36.209256281 +0000 UTC m=+2.860754822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls") pod "image-registry-5cf56cb474-gx24x" (UID: "877326a8-91a3-499f-8b37-4f80dfb00352") : secret "image-registry-tls" not found Oct 29 22:42:35.209553 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:35.209352 5057 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 22:42:35.209553 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:35.209411 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls podName:266bff54-6027-4822-91ce-be5cbd316ddc nodeName:}" failed. No retries permitted until 2026-10-29 22:42:36.20940049 +0000 UTC m=+2.860899005 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls") pod "dns-default-gj6vs" (UID: "266bff54-6027-4822-91ce-be5cbd316ddc") : secret "dns-default-metrics-tls" not found Oct 29 22:42:35.227908 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.227840 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-rwfmh"] Oct 29 22:42:35.232632 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:35.231844 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575ee3c4_7c77_4b31_9961_eebc60254d19.slice/crio-d8f96455aa2ee6509196546046a76f442c4800ee20cebf4e82b8523deca2435f WatchSource:0}: Error finding container d8f96455aa2ee6509196546046a76f442c4800ee20cebf4e82b8523deca2435f: Status 404 returned error can't find the container with id d8f96455aa2ee6509196546046a76f442c4800ee20cebf4e82b8523deca2435f Oct 29 22:42:35.400018 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.399986 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-zl99j_9483bf09-4676-47db-a279-84aabe9d17ed/migrator/0.log" Oct 29 22:42:35.541725 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.541647 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 22:42:35.542164 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.541757 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" event={"ID":"9e91a303-9b80-47f5-a502-fbb6365eaa54","Type":"ContainerStarted","Data":"209a3fe5ff66c7d05e0679fa946e39d73b388628202c564c48b54a0d52cfb5c9"} Oct 29 22:42:35.543321 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.543282 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7945df5b5-pcgp6" event={"ID":"99577b75-29cd-430b-b5ac-2af3828586d0","Type":"ContainerStarted","Data":"3c3b1e96e79cc27f5b1e5bfc45a27864d74c990143faa43087b78b2e3336a0fc"} Oct 29 22:42:35.543450 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.543330 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7945df5b5-pcgp6" event={"ID":"99577b75-29cd-430b-b5ac-2af3828586d0","Type":"ContainerStarted","Data":"874e2dc343e4c79c8132d99affde5881940223e66ee73b998a5914c1e76ccc66"} Oct 29 22:42:35.544684 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.544657 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" event={"ID":"96726184-9be2-4685-93d1-415bbb8b551c","Type":"ContainerStarted","Data":"01e5ebb06290031b5c89d6aec1abaf26637b9958f19b322f80bc37f7faded4d9"} Oct 29 22:42:35.546409 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.546376 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-rwfmh" event={"ID":"575ee3c4-7c77-4b31-9961-eebc60254d19","Type":"ContainerStarted","Data":"d8f96455aa2ee6509196546046a76f442c4800ee20cebf4e82b8523deca2435f"} Oct 29 22:42:35.547708 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.547674 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mxg8q" event={"ID":"286133c3-abe5-4954-83a1-8665c1785b92","Type":"ContainerStarted","Data":"ecb515a19180a745bdd6c1b4168a3b7d02c0e6de55911e4f43400ce1ed2822c1"} Oct 29 22:42:35.588849 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.588815 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-7klbh_093ceb90-294f-4133-83a6-32548405d530/kube-storage-version-migrator-operator/0.log" Oct 29 22:42:35.785478 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.785450 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:35.788290 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:35.788247 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:36.220926 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:36.220894 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:36.221109 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:36.220942 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:36.221109 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:36.221017 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:36.221109 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:36.221044 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:36.221109 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:36.221064 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:36.221109 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:36.221082 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:36.221423 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:36.221198 5057 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 22:42:36.221423 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:36.221257 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm podName:d518d30f-c46b-49ca-b74c-9f4cf0ca71e9 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:38.221238592 +0000 UTC m=+4.872737157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6rqj6" (UID: "d518d30f-c46b-49ca-b74c-9f4cf0ca71e9") : configmap "kube-rbac-proxy" not found Oct 29 22:42:36.222132 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:36.221626 5057 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:42:36.222132 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:36.221647 5057 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:42:36.222132 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:36.221743 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls podName:ad00084a-5a9f-489b-9122-2316f617f3f1 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:38.221693765 +0000 UTC m=+4.873192288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dzx6b" (UID: "ad00084a-5a9f-489b-9122-2316f617f3f1") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:42:36.222132 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:36.221766 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls podName:d518d30f-c46b-49ca-b74c-9f4cf0ca71e9 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:38.221755226 +0000 UTC m=+4.873253735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6rqj6" (UID: "d518d30f-c46b-49ca-b74c-9f4cf0ca71e9") : secret "insights-runtime-extractor-tls" not found Oct 29 22:42:36.222132 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:36.221778 5057 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 22:42:36.222132 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:36.221810 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls podName:266bff54-6027-4822-91ce-be5cbd316ddc nodeName:}" failed. No retries permitted until 2026-10-29 22:42:38.2218002 +0000 UTC m=+4.873298711 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls") pod "dns-default-gj6vs" (UID: "266bff54-6027-4822-91ce-be5cbd316ddc") : secret "dns-default-metrics-tls" not found Oct 29 22:42:36.224246 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:36.224225 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls\") pod \"image-registry-5cf56cb474-gx24x\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:36.224634 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:36.224614 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls\") pod \"image-registry-78f4dff4c9-87jff\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:36.285617 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:36.285579 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:36.298323 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:36.298290 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:36.746622 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:36.745403 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78f4dff4c9-87jff"] Oct 29 22:42:36.748298 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:36.748132 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cf56cb474-gx24x"] Oct 29 22:42:36.751383 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:36.751295 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884370ff_c54f_4d6b_af78_5c0813bb48eb.slice/crio-bebb3cbd9c055a012531957e405932ac1d63d38e0a667a1e9479ae336de49d9b WatchSource:0}: Error finding container bebb3cbd9c055a012531957e405932ac1d63d38e0a667a1e9479ae336de49d9b: Status 404 returned error can't find the container with id bebb3cbd9c055a012531957e405932ac1d63d38e0a667a1e9479ae336de49d9b Oct 29 22:42:36.759928 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:36.759863 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877326a8_91a3_499f_8b37_4f80dfb00352.slice/crio-9c1fa1bd0b03a99349d76396d8e66f9935453c20cacf5446adc321928ba34bbe WatchSource:0}: Error finding container 9c1fa1bd0b03a99349d76396d8e66f9935453c20cacf5446adc321928ba34bbe: Status 404 returned error can't find the container with id 9c1fa1bd0b03a99349d76396d8e66f9935453c20cacf5446adc321928ba34bbe Oct 29 22:42:37.559053 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:37.559016 5057 generic.go:356] "Generic (PLEG): container finished" podID="a109715c-c29a-48d1-a573-3e7800f48259" containerID="f4bf63c72257935d12452566cc01f3a0896e3340b4d39913d0ed28840ada515d" exitCode=0 Oct 29 22:42:37.559292 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:37.559088 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" event={"ID":"a109715c-c29a-48d1-a573-3e7800f48259","Type":"ContainerDied","Data":"f4bf63c72257935d12452566cc01f3a0896e3340b4d39913d0ed28840ada515d"} Oct 29 22:42:37.559519 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:37.559501 5057 scope.go:122] "RemoveContainer" containerID="f4bf63c72257935d12452566cc01f3a0896e3340b4d39913d0ed28840ada515d" Oct 29 22:42:37.560768 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:37.560734 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" event={"ID":"884370ff-c54f-4d6b-af78-5c0813bb48eb","Type":"ContainerStarted","Data":"d8ea0a970d4858bf67bb65495c9ec096d88995543a29e168bac67e26b4ec4315"} Oct 29 22:42:37.560873 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:37.560781 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" event={"ID":"884370ff-c54f-4d6b-af78-5c0813bb48eb","Type":"ContainerStarted","Data":"bebb3cbd9c055a012531957e405932ac1d63d38e0a667a1e9479ae336de49d9b"} Oct 29 22:42:37.562374 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:37.562353 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" event={"ID":"877326a8-91a3-499f-8b37-4f80dfb00352","Type":"ContainerStarted","Data":"5e51e0e1fac870b01fcb4a4631a5aef1b86065531948e703c2fe5748238a04ad"} Oct 29 22:42:37.562466 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:37.562395 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" event={"ID":"877326a8-91a3-499f-8b37-4f80dfb00352","Type":"ContainerStarted","Data":"9c1fa1bd0b03a99349d76396d8e66f9935453c20cacf5446adc321928ba34bbe"} Oct 29 22:42:37.564426 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:37.564404 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" event={"ID":"96726184-9be2-4685-93d1-415bbb8b551c","Type":"ContainerStarted","Data":"b6e92f37315c78164a7828224cbf8ecebba596e7b11d888acb47ad59cd31bce8"} Oct 29 22:42:37.564509 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:37.564451 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8ncgh" event={"ID":"96726184-9be2-4685-93d1-415bbb8b551c","Type":"ContainerStarted","Data":"7dbce71bf3f7a6fb20d6a690a0016dfebf5706e74cecb58731e9449dca88dd10"} Oct 29 22:42:37.565980 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:37.565956 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-rwfmh" event={"ID":"575ee3c4-7c77-4b31-9961-eebc60254d19","Type":"ContainerStarted","Data":"1a56ecc889be77fc906fba2e8067c1c15c90203c8563c50d2c9266a6027ceb0c"} Oct 29 22:42:37.567303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:37.567283 5057 generic.go:356] "Generic (PLEG): container finished" podID="093ceb90-294f-4133-83a6-32548405d530" containerID="45366dc34188d8f753af295ef018b066b8b533816d92e6b514a65863864e0d55" exitCode=0 Oct 29 22:42:37.567386 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:37.567350 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" event={"ID":"093ceb90-294f-4133-83a6-32548405d530","Type":"ContainerDied","Data":"45366dc34188d8f753af295ef018b066b8b533816d92e6b514a65863864e0d55"} Oct 29 22:42:37.567605 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:37.567587 5057 scope.go:122] "RemoveContainer" containerID="45366dc34188d8f753af295ef018b066b8b533816d92e6b514a65863864e0d55" Oct 29 22:42:37.571470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:37.571448 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mxg8q" event={"ID":"286133c3-abe5-4954-83a1-8665c1785b92","Type":"ContainerStarted","Data":"a81959de636f4a8d343e0c120c8da393f8cb031f3c0ddedf847938d796672a54"} Oct 29 22:42:37.571671 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:37.571475 5057 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:42:37.571671 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:37.571499 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mxg8q" event={"ID":"286133c3-abe5-4954-83a1-8665c1785b92","Type":"ContainerStarted","Data":"5ebf9be80ac1935185b37faf92dbfa216308a89afa5cc6e551cd8fd1a2b917f5"} Oct 29 22:42:38.242729 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:38.242691 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:38.242729 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:38.242733 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:38.243299 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:38.242756 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:38.243299 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:38.242775 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:38.243299 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:38.242843 5057 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 22:42:38.243299 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:38.242844 5057 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:42:38.243299 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:38.242896 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm podName:d518d30f-c46b-49ca-b74c-9f4cf0ca71e9 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:42.242882936 +0000 UTC m=+8.894381444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6rqj6" (UID: "d518d30f-c46b-49ca-b74c-9f4cf0ca71e9") : configmap "kube-rbac-proxy" not found Oct 29 22:42:38.243299 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:38.242843 5057 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:42:38.243299 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:38.243015 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls podName:ad00084a-5a9f-489b-9122-2316f617f3f1 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:42.242974626 +0000 UTC m=+8.894473164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dzx6b" (UID: "ad00084a-5a9f-489b-9122-2316f617f3f1") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:42:38.243299 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:42:38.243162 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls podName:d518d30f-c46b-49ca-b74c-9f4cf0ca71e9 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:42.243091624 +0000 UTC m=+8.894590147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6rqj6" (UID: "d518d30f-c46b-49ca-b74c-9f4cf0ca71e9") : secret "insights-runtime-extractor-tls" not found Oct 29 22:42:38.245116 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:38.245094 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266bff54-6027-4822-91ce-be5cbd316ddc-metrics-tls\") pod \"dns-default-gj6vs\" (UID: \"266bff54-6027-4822-91ce-be5cbd316ddc\") " pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:38.340859 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:38.340824 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:38.341004 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:38.340981 5057 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:42:38.345810 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:38.345784 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-lm55w" Oct 29 22:42:38.398949 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:38.398914 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:38.576994 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:38.576910 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j74gf" event={"ID":"a109715c-c29a-48d1-a573-3e7800f48259","Type":"ContainerStarted","Data":"b86ffcb551e293d8bac27568c29a09cadd573e825cad39ce92509a8c9cf73661"} Oct 29 22:42:38.578754 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:38.578733 5057 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:42:38.578877 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:38.578772 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7klbh" event={"ID":"093ceb90-294f-4133-83a6-32548405d530","Type":"ContainerStarted","Data":"66a19b72290358573ff8a436d8dc07eb7b05256917f29bdcc9ab795530ccc8d2"} Oct 29 22:42:38.579085 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:38.579067 5057 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:42:38.697446 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:38.697423 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gj6vs"] Oct 29 22:42:38.699381 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:38.699350 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266bff54_6027_4822_91ce_be5cbd316ddc.slice/crio-88f5d2ee95780dd5dba734f898c92190a446ba18742789110250b5120439f5d6 WatchSource:0}: Error finding container 88f5d2ee95780dd5dba734f898c92190a446ba18742789110250b5120439f5d6: Status 404 returned error can't find the container with id 88f5d2ee95780dd5dba734f898c92190a446ba18742789110250b5120439f5d6 Oct 29 22:42:39.584212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:39.584163 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gj6vs" event={"ID":"266bff54-6027-4822-91ce-be5cbd316ddc","Type":"ContainerStarted","Data":"88f5d2ee95780dd5dba734f898c92190a446ba18742789110250b5120439f5d6"} Oct 29 22:42:40.590441 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:40.590399 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gj6vs" event={"ID":"266bff54-6027-4822-91ce-be5cbd316ddc","Type":"ContainerStarted","Data":"0f75580cfdef3100e28b656ccbeafb225c0a369f718c93ca56fb2df1fbc5d6d9"} Oct 29 22:42:40.590814 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:40.590477 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gj6vs" event={"ID":"266bff54-6027-4822-91ce-be5cbd316ddc","Type":"ContainerStarted","Data":"c5572cbc535eaa7288bdb8a1bb32a4be3a2d73aa10f99152a19c60843ccfce3c"} Oct 29 22:42:40.590814 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:40.590527 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:41.303224 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:41.303168 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:42:42.226463 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:42.226422 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:42.281292 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:42.281255 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:42.281423 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:42.281302 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:42.281423 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:42.281334 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:42.281973 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:42.281946 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:42.283656 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:42.283634 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d518d30f-c46b-49ca-b74c-9f4cf0ca71e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6rqj6\" (UID: \"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9\") " pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:42.283755 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:42.283665 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad00084a-5a9f-489b-9122-2316f617f3f1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dzx6b\" (UID: \"ad00084a-5a9f-489b-9122-2316f617f3f1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:42.300750 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:42.300729 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6rqj6" Oct 29 22:42:42.378385 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:42.378360 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:42.378650 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:42.378635 5057 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:42:42.583908 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:42.583830 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" Oct 29 22:42:42.596181 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:42.596157 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6rqj6"] Oct 29 22:42:42.598169 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:42.598140 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd518d30f_c46b_49ca_b74c_9f4cf0ca71e9.slice/crio-4063931122b1965999f467b0697ee63f3b1d017336f9758836dc5cf31c4a66a4 WatchSource:0}: Error finding container 4063931122b1965999f467b0697ee63f3b1d017336f9758836dc5cf31c4a66a4: Status 404 returned error can't find the container with id 4063931122b1965999f467b0697ee63f3b1d017336f9758836dc5cf31c4a66a4 Oct 29 22:42:42.887154 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:42.887129 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b"] Oct 29 22:42:42.889563 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:42.889534 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad00084a_5a9f_489b_9122_2316f617f3f1.slice/crio-93a2eb2f92bf5abaa38173b7153787cd95ae66b005b23ab226317a2b01a79961 WatchSource:0}: Error finding container 93a2eb2f92bf5abaa38173b7153787cd95ae66b005b23ab226317a2b01a79961: Status 404 returned error can't find the container with id 93a2eb2f92bf5abaa38173b7153787cd95ae66b005b23ab226317a2b01a79961 Oct 29 22:42:43.182751 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:43.182728 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:43.182888 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:43.182873 5057 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:42:43.184784 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:43.184763 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7945df5b5-pcgp6" Oct 29 22:42:43.605085 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:43.604996 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6rqj6" event={"ID":"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9","Type":"ContainerStarted","Data":"cb8830936af1a4c1180d8b77170e13b8e926d979fa1fb7a47771299256b5df5b"} Oct 29 22:42:43.605527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:43.605120 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6rqj6" event={"ID":"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9","Type":"ContainerStarted","Data":"4f0043a839e2f0a08e0350690909b81b1e23698599fdb367bf2f70f249e5d435"} Oct 29 22:42:43.605527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:43.605149 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6rqj6" event={"ID":"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9","Type":"ContainerStarted","Data":"4063931122b1965999f467b0697ee63f3b1d017336f9758836dc5cf31c4a66a4"} Oct 29 22:42:43.606347 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:43.606318 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" event={"ID":"ad00084a-5a9f-489b-9122-2316f617f3f1","Type":"ContainerStarted","Data":"93a2eb2f92bf5abaa38173b7153787cd95ae66b005b23ab226317a2b01a79961"} Oct 29 22:42:44.388963 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:44.388932 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-99ngw" Oct 29 22:42:44.613163 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:44.613129 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dzx6b" event={"ID":"ad00084a-5a9f-489b-9122-2316f617f3f1","Type":"ContainerStarted","Data":"9fabcfa1f18af2381fa4cafdfeca688ced2f883a8122ef9873a1180d3637bc82"} Oct 29 22:42:45.619397 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:45.619357 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6rqj6" event={"ID":"d518d30f-c46b-49ca-b74c-9f4cf0ca71e9","Type":"ContainerStarted","Data":"4b4da3510ef079667c0e586cf240b1a3d491b05f5e8199c3127292bb044fadec"} Oct 29 22:42:46.080520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:46.080479 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:42:48.161666 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.161632 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-cbgxh"] Oct 29 22:42:48.164521 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.164489 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-cbgxh" Oct 29 22:42:48.174608 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.174579 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d8bc9f8cf-db28b"] Oct 29 22:42:48.176835 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.176813 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.184130 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.183929 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cf56cb474-gx24x"] Oct 29 22:42:48.195205 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.195161 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 29 22:42:48.195403 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.195176 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 29 22:42:48.195499 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.195273 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-7cnf2\"" Oct 29 22:42:48.196273 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.196242 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d8bc9f8cf-db28b"] Oct 29 22:42:48.196996 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.196972 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-cbgxh"] Oct 29 22:42:48.202526 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.202504 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 29 22:42:48.202642 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.202536 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 29 22:42:48.202642 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.202553 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 29 22:42:48.245426 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.245383 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-582ld"] Oct 29 22:42:48.248766 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.248743 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-585b5456bf-pk6zv"] Oct 29 22:42:48.248898 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.248864 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-582ld" Oct 29 22:42:48.250944 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.250924 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.275034 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.275009 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-582ld"] Oct 29 22:42:48.282598 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.282567 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-585b5456bf-pk6zv"] Oct 29 22:42:48.286642 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.286617 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 29 22:42:48.286642 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.286633 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 29 22:42:48.286799 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.286661 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 29 22:42:48.286799 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.286678 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-q5gh9\"" Oct 29 22:42:48.286799 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.286692 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dxz6f\"" Oct 29 22:42:48.329903 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.329877 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-tmp\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.330061 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.329916 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-oauth-serving-cert\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.330061 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.329944 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-service-ca\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.330061 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.330028 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-config\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.330061 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.330055 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-serving-cert\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.330238 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.330097 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfdg2\" (UniqueName: \"kubernetes.io/projected/cb6f2831-3e8c-440c-a5da-327ab5cf947b-kube-api-access-vfdg2\") pod \"downloads-6dd789c8b7-cbgxh\" (UID: \"cb6f2831-3e8c-440c-a5da-327ab5cf947b\") " pod="openshift-console/downloads-6dd789c8b7-cbgxh" Oct 29 22:42:48.330238 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.330130 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb6f2831-3e8c-440c-a5da-327ab5cf947b-tmp\") pod \"downloads-6dd789c8b7-cbgxh\" (UID: \"cb6f2831-3e8c-440c-a5da-327ab5cf947b\") " pod="openshift-console/downloads-6dd789c8b7-cbgxh" Oct 29 22:42:48.330238 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.330157 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67vrn\" (UniqueName: \"kubernetes.io/projected/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-kube-api-access-67vrn\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.330238 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.330182 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-oauth-config\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.430539 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.430500 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2jfx\" (UniqueName: \"kubernetes.io/projected/33110848-7553-4685-b74a-08ef49fa172e-kube-api-access-n2jfx\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.430736 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.430548 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-serving-cert\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.430736 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.430570 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33110848-7553-4685-b74a-08ef49fa172e-registry-certificates\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.430736 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.430638 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfdg2\" (UniqueName: \"kubernetes.io/projected/cb6f2831-3e8c-440c-a5da-327ab5cf947b-kube-api-access-vfdg2\") pod \"downloads-6dd789c8b7-cbgxh\" (UID: \"cb6f2831-3e8c-440c-a5da-327ab5cf947b\") " pod="openshift-console/downloads-6dd789c8b7-cbgxh" Oct 29 22:42:48.430736 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.430676 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb6f2831-3e8c-440c-a5da-327ab5cf947b-tmp\") pod \"downloads-6dd789c8b7-cbgxh\" (UID: \"cb6f2831-3e8c-440c-a5da-327ab5cf947b\") " pod="openshift-console/downloads-6dd789c8b7-cbgxh" Oct 29 22:42:48.430736 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.430704 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33110848-7553-4685-b74a-08ef49fa172e-trusted-ca\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.430736 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.430722 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33110848-7553-4685-b74a-08ef49fa172e-registry-tls\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.431032 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.430742 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67vrn\" (UniqueName: \"kubernetes.io/projected/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-kube-api-access-67vrn\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.431032 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.430770 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-oauth-config\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.431032 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.430799 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33110848-7553-4685-b74a-08ef49fa172e-ca-trust-extracted\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.431032 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.430916 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-tmp\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.431032 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.430954 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33110848-7553-4685-b74a-08ef49fa172e-bound-sa-token\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.431032 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.430986 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/16d0a575-91ff-47c7-ab38-a62c542e4c9c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-582ld\" (UID: \"16d0a575-91ff-47c7-ab38-a62c542e4c9c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-582ld" Oct 29 22:42:48.431332 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.431094 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-oauth-serving-cert\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.431332 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.431146 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33110848-7553-4685-b74a-08ef49fa172e-installation-pull-secrets\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.431332 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.431173 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb6f2831-3e8c-440c-a5da-327ab5cf947b-tmp\") pod \"downloads-6dd789c8b7-cbgxh\" (UID: \"cb6f2831-3e8c-440c-a5da-327ab5cf947b\") " pod="openshift-console/downloads-6dd789c8b7-cbgxh" Oct 29 22:42:48.431332 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.431208 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-service-ca\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.431332 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.431269 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33110848-7553-4685-b74a-08ef49fa172e-image-registry-private-configuration\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.431332 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.431300 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-tmp\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.431332 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.431303 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-config\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.431957 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.431930 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-service-ca\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.432073 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.431958 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-oauth-serving-cert\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.432073 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.431974 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-config\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.433314 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.433295 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-serving-cert\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.433397 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.433323 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-oauth-config\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.459970 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.459945 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfdg2\" (UniqueName: \"kubernetes.io/projected/cb6f2831-3e8c-440c-a5da-327ab5cf947b-kube-api-access-vfdg2\") pod \"downloads-6dd789c8b7-cbgxh\" (UID: \"cb6f2831-3e8c-440c-a5da-327ab5cf947b\") " pod="openshift-console/downloads-6dd789c8b7-cbgxh" Oct 29 22:42:48.462941 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.462915 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67vrn\" (UniqueName: \"kubernetes.io/projected/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-kube-api-access-67vrn\") pod \"console-7d8bc9f8cf-db28b\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.476843 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.476818 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-cbgxh" Oct 29 22:42:48.491690 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.491660 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:48.532286 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.532221 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33110848-7553-4685-b74a-08ef49fa172e-installation-pull-secrets\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.532521 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.532462 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33110848-7553-4685-b74a-08ef49fa172e-image-registry-private-configuration\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.532760 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.532726 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2jfx\" (UniqueName: \"kubernetes.io/projected/33110848-7553-4685-b74a-08ef49fa172e-kube-api-access-n2jfx\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.532935 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.532860 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33110848-7553-4685-b74a-08ef49fa172e-registry-certificates\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.533074 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.533007 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33110848-7553-4685-b74a-08ef49fa172e-trusted-ca\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.533074 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.533027 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33110848-7553-4685-b74a-08ef49fa172e-registry-tls\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.534051 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.533667 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33110848-7553-4685-b74a-08ef49fa172e-ca-trust-extracted\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.534051 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.533707 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33110848-7553-4685-b74a-08ef49fa172e-ca-trust-extracted\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.534051 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.533741 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33110848-7553-4685-b74a-08ef49fa172e-bound-sa-token\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.534051 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.533760 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/16d0a575-91ff-47c7-ab38-a62c542e4c9c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-582ld\" (UID: \"16d0a575-91ff-47c7-ab38-a62c542e4c9c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-582ld" Oct 29 22:42:48.536347 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.536299 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33110848-7553-4685-b74a-08ef49fa172e-trusted-ca\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.537653 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.537632 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33110848-7553-4685-b74a-08ef49fa172e-registry-certificates\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.537778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.537666 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33110848-7553-4685-b74a-08ef49fa172e-installation-pull-secrets\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.538695 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.538640 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33110848-7553-4685-b74a-08ef49fa172e-image-registry-private-configuration\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.539996 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.539967 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33110848-7553-4685-b74a-08ef49fa172e-registry-tls\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.541085 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.541060 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/16d0a575-91ff-47c7-ab38-a62c542e4c9c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-582ld\" (UID: \"16d0a575-91ff-47c7-ab38-a62c542e4c9c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-582ld" Oct 29 22:42:48.560768 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.560679 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-582ld" Oct 29 22:42:48.562426 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.562399 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33110848-7553-4685-b74a-08ef49fa172e-bound-sa-token\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.562981 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.562956 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2jfx\" (UniqueName: \"kubernetes.io/projected/33110848-7553-4685-b74a-08ef49fa172e-kube-api-access-n2jfx\") pod \"image-registry-585b5456bf-pk6zv\" (UID: \"33110848-7553-4685-b74a-08ef49fa172e\") " pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.567892 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.567506 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:48.792857 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.792827 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-cbgxh"] Oct 29 22:42:48.794725 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:48.794697 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6f2831_3e8c_440c_a5da_327ab5cf947b.slice/crio-cf6c800d1e08e2e5c20fa1029411e9cf9749fad0aaee66c000d7119b946434af WatchSource:0}: Error finding container cf6c800d1e08e2e5c20fa1029411e9cf9749fad0aaee66c000d7119b946434af: Status 404 returned error can't find the container with id cf6c800d1e08e2e5c20fa1029411e9cf9749fad0aaee66c000d7119b946434af Oct 29 22:42:48.839488 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.839453 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d8bc9f8cf-db28b"] Oct 29 22:42:48.845714 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:48.845687 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4841ad9_20fe_48cf_83d7_a6b95d4992ed.slice/crio-b40b0bc67d568263f18df0f1cb74b98517308dbe742d9410e3fd173b3559100b WatchSource:0}: Error finding container b40b0bc67d568263f18df0f1cb74b98517308dbe742d9410e3fd173b3559100b: Status 404 returned error can't find the container with id b40b0bc67d568263f18df0f1cb74b98517308dbe742d9410e3fd173b3559100b Oct 29 22:42:48.886645 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.886611 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-585b5456bf-pk6zv"] Oct 29 22:42:48.887877 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:48.887851 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-582ld"] Oct 29 22:42:48.888551 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:48.888510 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d0a575_91ff_47c7_ab38_a62c542e4c9c.slice/crio-d0a1f3f7ecfd1046f180502fc0556472e12a727f51c4a18c5461ce35b62bf34e WatchSource:0}: Error finding container d0a1f3f7ecfd1046f180502fc0556472e12a727f51c4a18c5461ce35b62bf34e: Status 404 returned error can't find the container with id d0a1f3f7ecfd1046f180502fc0556472e12a727f51c4a18c5461ce35b62bf34e Oct 29 22:42:48.891109 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:48.891088 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33110848_7553_4685_b74a_08ef49fa172e.slice/crio-b960360b232389205ea26bb0514870eba480080aa1e4bef68ff724606ec3d34f WatchSource:0}: Error finding container b960360b232389205ea26bb0514870eba480080aa1e4bef68ff724606ec3d34f: Status 404 returned error can't find the container with id b960360b232389205ea26bb0514870eba480080aa1e4bef68ff724606ec3d34f Oct 29 22:42:49.642662 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:49.642585 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d8bc9f8cf-db28b" event={"ID":"e4841ad9-20fe-48cf-83d7-a6b95d4992ed","Type":"ContainerStarted","Data":"b40b0bc67d568263f18df0f1cb74b98517308dbe742d9410e3fd173b3559100b"} Oct 29 22:42:49.644410 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:49.644343 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-582ld" event={"ID":"16d0a575-91ff-47c7-ab38-a62c542e4c9c","Type":"ContainerStarted","Data":"d0a1f3f7ecfd1046f180502fc0556472e12a727f51c4a18c5461ce35b62bf34e"} Oct 29 22:42:49.646960 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:49.646009 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" event={"ID":"33110848-7553-4685-b74a-08ef49fa172e","Type":"ContainerStarted","Data":"28f8fc6d32a677cb0a79226539d7b4286417dd319c2f1cbf5577ae4a48a3c644"} Oct 29 22:42:49.646960 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:49.646051 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" event={"ID":"33110848-7553-4685-b74a-08ef49fa172e","Type":"ContainerStarted","Data":"b960360b232389205ea26bb0514870eba480080aa1e4bef68ff724606ec3d34f"} Oct 29 22:42:49.646960 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:49.646904 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:42:49.648956 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:49.648932 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-cbgxh" event={"ID":"cb6f2831-3e8c-440c-a5da-327ab5cf947b","Type":"ContainerStarted","Data":"cf6c800d1e08e2e5c20fa1029411e9cf9749fad0aaee66c000d7119b946434af"} Oct 29 22:42:49.695819 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:49.694221 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" podStartSLOduration=1.694201605 podStartE2EDuration="1.694201605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:49.693851168 +0000 UTC m=+16.345349699" watchObservedRunningTime="2026-10-29 22:42:49.694201605 +0000 UTC m=+16.345700127" Oct 29 22:42:50.596445 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:50.596411 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gj6vs" Oct 29 22:42:50.659283 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:50.659242 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-582ld" event={"ID":"16d0a575-91ff-47c7-ab38-a62c542e4c9c","Type":"ContainerStarted","Data":"7253ca19711871385a7acef1a50d1be60c41ea2aae418dd360b2645c9909d708"} Oct 29 22:42:50.659725 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:50.659571 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-582ld" Oct 29 22:42:50.665122 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:50.665089 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-582ld" Oct 29 22:42:50.703516 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:50.702274 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-582ld" podStartSLOduration=1.6129980800000001 podStartE2EDuration="2.702254385s" totalImagesPullingTime="1.089256305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:50.701938616 +0000 UTC m=+17.353437147" watchObservedRunningTime="2026-10-29 22:42:50.702254385 +0000 UTC m=+17.353752912" Oct 29 22:42:51.071589 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:51.071544 5057 patch_prober.go:29] interesting pod/image-registry-78f4dff4c9-87jff container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 29 22:42:51.071781 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:51.071604 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" podUID="884370ff-c54f-4d6b-af78-5c0813bb48eb" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:42:52.673010 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:52.672976 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d8bc9f8cf-db28b" event={"ID":"e4841ad9-20fe-48cf-83d7-a6b95d4992ed","Type":"ContainerStarted","Data":"4b8d2b20b37327ec33c095ca826bf92febc27e1c12ea51fb2547e50e87b94d24"} Oct 29 22:42:52.719856 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:52.719796 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7d8bc9f8cf-db28b" podStartSLOduration=1.6523547509999998 podStartE2EDuration="4.719773802s" totalImagesPullingTime="3.067419051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:52.718987751 +0000 UTC m=+19.370486293" watchObservedRunningTime="2026-10-29 22:42:52.719773802 +0000 UTC m=+19.371272333" Oct 29 22:42:56.086605 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.086548 5057 patch_prober.go:29] interesting pod/image-registry-78f4dff4c9-87jff container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 29 22:42:56.087095 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.086614 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" podUID="884370ff-c54f-4d6b-af78-5c0813bb48eb" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:42:56.222353 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.222316 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mrtf7"] Oct 29 22:42:56.226057 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.226037 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.291757 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.291699 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 29 22:42:56.292304 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.292272 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x4fl8\"" Oct 29 22:42:56.292409 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.292360 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 29 22:42:56.304378 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.304348 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 29 22:42:56.304572 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.304553 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 29 22:42:56.308070 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.308032 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45hch\" (UniqueName: \"kubernetes.io/projected/225b1337-fa33-4932-97c0-bbad4f83aab4-kube-api-access-45hch\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.308169 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.308157 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/225b1337-fa33-4932-97c0-bbad4f83aab4-sys\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.308326 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.308206 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/225b1337-fa33-4932-97c0-bbad4f83aab4-node-exporter-textfile\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.308506 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.308349 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/225b1337-fa33-4932-97c0-bbad4f83aab4-node-exporter-accelerators-collector-config\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.308592 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.308576 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/225b1337-fa33-4932-97c0-bbad4f83aab4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.308742 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.308728 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/225b1337-fa33-4932-97c0-bbad4f83aab4-root\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.308809 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.308795 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/225b1337-fa33-4932-97c0-bbad4f83aab4-node-exporter-tls\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.308852 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.308828 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/225b1337-fa33-4932-97c0-bbad4f83aab4-node-exporter-wtmp\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.308889 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.308873 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/225b1337-fa33-4932-97c0-bbad4f83aab4-metrics-client-ca\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.409578 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.409491 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/225b1337-fa33-4932-97c0-bbad4f83aab4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.409578 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.409558 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/225b1337-fa33-4932-97c0-bbad4f83aab4-root\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.409805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.409605 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/225b1337-fa33-4932-97c0-bbad4f83aab4-node-exporter-tls\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.409805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.409622 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/225b1337-fa33-4932-97c0-bbad4f83aab4-node-exporter-wtmp\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.409805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.409648 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/225b1337-fa33-4932-97c0-bbad4f83aab4-metrics-client-ca\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.409805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.409671 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/225b1337-fa33-4932-97c0-bbad4f83aab4-root\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.409805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.409686 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45hch\" (UniqueName: \"kubernetes.io/projected/225b1337-fa33-4932-97c0-bbad4f83aab4-kube-api-access-45hch\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.409805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.409736 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/225b1337-fa33-4932-97c0-bbad4f83aab4-sys\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.409805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.409763 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/225b1337-fa33-4932-97c0-bbad4f83aab4-node-exporter-textfile\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.409805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.409792 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/225b1337-fa33-4932-97c0-bbad4f83aab4-node-exporter-accelerators-collector-config\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.410306 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.409836 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/225b1337-fa33-4932-97c0-bbad4f83aab4-node-exporter-wtmp\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.410306 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.409903 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/225b1337-fa33-4932-97c0-bbad4f83aab4-sys\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.410306 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.410153 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/225b1337-fa33-4932-97c0-bbad4f83aab4-node-exporter-textfile\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.410306 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.410269 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/225b1337-fa33-4932-97c0-bbad4f83aab4-metrics-client-ca\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.410524 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.410336 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/225b1337-fa33-4932-97c0-bbad4f83aab4-node-exporter-accelerators-collector-config\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.412254 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.412232 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/225b1337-fa33-4932-97c0-bbad4f83aab4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.414822 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.414795 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/225b1337-fa33-4932-97c0-bbad4f83aab4-node-exporter-tls\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.434330 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.434302 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45hch\" (UniqueName: \"kubernetes.io/projected/225b1337-fa33-4932-97c0-bbad4f83aab4-kube-api-access-45hch\") pod \"node-exporter-mrtf7\" (UID: \"225b1337-fa33-4932-97c0-bbad4f83aab4\") " pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.542496 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.542460 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mrtf7" Oct 29 22:42:56.554914 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:42:56.554881 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225b1337_fa33_4932_97c0_bbad4f83aab4.slice/crio-f8e820ab2eb9b32d44d13092eac16f75711cb030f55ba37582851d7b1e9a569b WatchSource:0}: Error finding container f8e820ab2eb9b32d44d13092eac16f75711cb030f55ba37582851d7b1e9a569b: Status 404 returned error can't find the container with id f8e820ab2eb9b32d44d13092eac16f75711cb030f55ba37582851d7b1e9a569b Oct 29 22:42:56.694740 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:56.694699 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mrtf7" event={"ID":"225b1337-fa33-4932-97c0-bbad4f83aab4","Type":"ContainerStarted","Data":"f8e820ab2eb9b32d44d13092eac16f75711cb030f55ba37582851d7b1e9a569b"} Oct 29 22:42:58.190954 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:58.190911 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:42:58.492821 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:58.492737 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:58.492821 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:58.492810 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:58.497922 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:58.497899 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:58.707006 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:58.706970 5057 generic.go:356] "Generic (PLEG): container finished" podID="225b1337-fa33-4932-97c0-bbad4f83aab4" containerID="016cde2225872e54abb19a236a287cabc7daca3d48102a4661c66e12d52aeed0" exitCode=0 Oct 29 22:42:58.707222 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:58.707065 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mrtf7" event={"ID":"225b1337-fa33-4932-97c0-bbad4f83aab4","Type":"ContainerDied","Data":"016cde2225872e54abb19a236a287cabc7daca3d48102a4661c66e12d52aeed0"} Oct 29 22:42:58.711636 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:58.711615 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:42:59.714847 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:59.714807 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mrtf7" event={"ID":"225b1337-fa33-4932-97c0-bbad4f83aab4","Type":"ContainerStarted","Data":"ba4c8fc605a124bccabd589844261b2859f51c72a0e820efe5c3243ab939197f"} Oct 29 22:42:59.715339 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:59.714910 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mrtf7" event={"ID":"225b1337-fa33-4932-97c0-bbad4f83aab4","Type":"ContainerStarted","Data":"b3b8e9c6de64a6685acf3fcd71ab83878b568b5736b4a29cf20ce25575975594"} Oct 29 22:42:59.786434 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:42:59.786366 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mrtf7" podStartSLOduration=2.409143376 podStartE2EDuration="3.786347153s" totalImagesPullingTime="1.289999907s" totalInitContainerRuntime="87.20387ms" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:59.784665216 +0000 UTC m=+26.436163746" watchObservedRunningTime="2026-10-29 22:42:59.786347153 +0000 UTC m=+26.437845687" Oct 29 22:43:00.384931 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:00.384900 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sc55n" Oct 29 22:43:01.307117 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:01.307084 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j8fft" Oct 29 22:43:03.916971 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:43:03.916932 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:43:05.754838 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:05.754796 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-cbgxh" event={"ID":"cb6f2831-3e8c-440c-a5da-327ab5cf947b","Type":"ContainerStarted","Data":"a39d6b7b9b50ead19d0454d8d2a109407309308c041a1279f689d5f9402ae613"} Oct 29 22:43:05.755338 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:05.755077 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-cbgxh" Oct 29 22:43:05.772716 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:05.772682 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-cbgxh" Oct 29 22:43:05.810673 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:05.810609 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-cbgxh" podStartSLOduration=1.363402086 podStartE2EDuration="17.810594314s" totalImagesPullingTime="16.447192228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:05.8101764 +0000 UTC m=+32.461674929" watchObservedRunningTime="2026-10-29 22:43:05.810594314 +0000 UTC m=+32.462092845" Oct 29 22:43:06.085785 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:06.085711 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:43:11.669679 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:11.669646 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-585b5456bf-pk6zv" Oct 29 22:43:11.764483 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:11.764445 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78f4dff4c9-87jff"] Oct 29 22:43:13.208724 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.208656 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" podUID="877326a8-91a3-499f-8b37-4f80dfb00352" containerName="registry" containerID="cri-o://5e51e0e1fac870b01fcb4a4631a5aef1b86065531948e703c2fe5748238a04ad" gracePeriod=30 Oct 29 22:43:13.523231 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.523175 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:43:13.680461 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.680425 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-kube-api-access-hcfvz\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-kube-api-access-hcfvz\") pod \"877326a8-91a3-499f-8b37-4f80dfb00352\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " Oct 29 22:43:13.680636 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.680480 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877326a8-91a3-499f-8b37-4f80dfb00352-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/877326a8-91a3-499f-8b37-4f80dfb00352-ca-trust-extracted\") pod \"877326a8-91a3-499f-8b37-4f80dfb00352\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " Oct 29 22:43:13.680636 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.680513 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/877326a8-91a3-499f-8b37-4f80dfb00352-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/877326a8-91a3-499f-8b37-4f80dfb00352-trusted-ca\") pod \"877326a8-91a3-499f-8b37-4f80dfb00352\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " Oct 29 22:43:13.680636 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.680545 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/877326a8-91a3-499f-8b37-4f80dfb00352-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/877326a8-91a3-499f-8b37-4f80dfb00352-registry-certificates\") pod \"877326a8-91a3-499f-8b37-4f80dfb00352\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " Oct 29 22:43:13.680636 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.680589 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/877326a8-91a3-499f-8b37-4f80dfb00352-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/877326a8-91a3-499f-8b37-4f80dfb00352-image-registry-private-configuration\") pod \"877326a8-91a3-499f-8b37-4f80dfb00352\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " Oct 29 22:43:13.680853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.680658 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/877326a8-91a3-499f-8b37-4f80dfb00352-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/877326a8-91a3-499f-8b37-4f80dfb00352-installation-pull-secrets\") pod \"877326a8-91a3-499f-8b37-4f80dfb00352\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " Oct 29 22:43:13.680853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.680716 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls\") pod \"877326a8-91a3-499f-8b37-4f80dfb00352\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " Oct 29 22:43:13.680853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.680749 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-bound-sa-token\") pod \"877326a8-91a3-499f-8b37-4f80dfb00352\" (UID: \"877326a8-91a3-499f-8b37-4f80dfb00352\") " Oct 29 22:43:13.681061 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.680984 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/877326a8-91a3-499f-8b37-4f80dfb00352-trusted-ca" pod "877326a8-91a3-499f-8b37-4f80dfb00352" (UID: "877326a8-91a3-499f-8b37-4f80dfb00352"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:13.681127 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.681086 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/877326a8-91a3-499f-8b37-4f80dfb00352-registry-certificates" pod "877326a8-91a3-499f-8b37-4f80dfb00352" (UID: "877326a8-91a3-499f-8b37-4f80dfb00352"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:13.681127 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.681109 5057 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/877326a8-91a3-499f-8b37-4f80dfb00352-trusted-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:13.684764 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.684715 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-kube-api-access-hcfvz" pod "877326a8-91a3-499f-8b37-4f80dfb00352" (UID: "877326a8-91a3-499f-8b37-4f80dfb00352"). InnerVolumeSpecName "kube-api-access-hcfvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:13.684945 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.684912 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls" pod "877326a8-91a3-499f-8b37-4f80dfb00352" (UID: "877326a8-91a3-499f-8b37-4f80dfb00352"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:13.685084 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.685059 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/877326a8-91a3-499f-8b37-4f80dfb00352-image-registry-private-configuration" pod "877326a8-91a3-499f-8b37-4f80dfb00352" (UID: "877326a8-91a3-499f-8b37-4f80dfb00352"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:13.687693 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.687656 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-bound-sa-token" pod "877326a8-91a3-499f-8b37-4f80dfb00352" (UID: "877326a8-91a3-499f-8b37-4f80dfb00352"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:13.689455 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.689372 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/877326a8-91a3-499f-8b37-4f80dfb00352-installation-pull-secrets" pod "877326a8-91a3-499f-8b37-4f80dfb00352" (UID: "877326a8-91a3-499f-8b37-4f80dfb00352"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:13.690039 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.690010 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877326a8-91a3-499f-8b37-4f80dfb00352-ca-trust-extracted" pod "877326a8-91a3-499f-8b37-4f80dfb00352" (UID: "877326a8-91a3-499f-8b37-4f80dfb00352"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:13.782096 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.782023 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcfvz\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-kube-api-access-hcfvz\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:13.782096 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.782050 5057 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/877326a8-91a3-499f-8b37-4f80dfb00352-ca-trust-extracted\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:13.782096 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.782059 5057 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/877326a8-91a3-499f-8b37-4f80dfb00352-registry-certificates\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:13.782096 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.782069 5057 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/877326a8-91a3-499f-8b37-4f80dfb00352-image-registry-private-configuration\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:13.782096 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.782080 5057 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/877326a8-91a3-499f-8b37-4f80dfb00352-installation-pull-secrets\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:13.782096 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.782089 5057 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-registry-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:13.782096 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.782098 5057 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/877326a8-91a3-499f-8b37-4f80dfb00352-bound-sa-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:13.795961 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.795923 5057 generic.go:356] "Generic (PLEG): container finished" podID="877326a8-91a3-499f-8b37-4f80dfb00352" containerID="5e51e0e1fac870b01fcb4a4631a5aef1b86065531948e703c2fe5748238a04ad" exitCode=0 Oct 29 22:43:13.796140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.795980 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" event={"ID":"877326a8-91a3-499f-8b37-4f80dfb00352","Type":"ContainerDied","Data":"5e51e0e1fac870b01fcb4a4631a5aef1b86065531948e703c2fe5748238a04ad"} Oct 29 22:43:13.796140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.796065 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" event={"ID":"877326a8-91a3-499f-8b37-4f80dfb00352","Type":"ContainerDied","Data":"9c1fa1bd0b03a99349d76396d8e66f9935453c20cacf5446adc321928ba34bbe"} Oct 29 22:43:13.796140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.796070 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf56cb474-gx24x" Oct 29 22:43:13.796140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.796087 5057 scope.go:122] "RemoveContainer" containerID="5e51e0e1fac870b01fcb4a4631a5aef1b86065531948e703c2fe5748238a04ad" Oct 29 22:43:13.806915 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.806894 5057 scope.go:122] "RemoveContainer" containerID="5e51e0e1fac870b01fcb4a4631a5aef1b86065531948e703c2fe5748238a04ad" Oct 29 22:43:13.807274 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:43:13.807242 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e51e0e1fac870b01fcb4a4631a5aef1b86065531948e703c2fe5748238a04ad\": container with ID starting with 5e51e0e1fac870b01fcb4a4631a5aef1b86065531948e703c2fe5748238a04ad not found: ID does not exist" containerID="5e51e0e1fac870b01fcb4a4631a5aef1b86065531948e703c2fe5748238a04ad" Oct 29 22:43:13.807366 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.807285 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e51e0e1fac870b01fcb4a4631a5aef1b86065531948e703c2fe5748238a04ad"} err="failed to get container status \"5e51e0e1fac870b01fcb4a4631a5aef1b86065531948e703c2fe5748238a04ad\": rpc error: code = NotFound desc = could not find container \"5e51e0e1fac870b01fcb4a4631a5aef1b86065531948e703c2fe5748238a04ad\": container with ID starting with 5e51e0e1fac870b01fcb4a4631a5aef1b86065531948e703c2fe5748238a04ad not found: ID does not exist" Oct 29 22:43:13.859746 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.859709 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cf56cb474-gx24x"] Oct 29 22:43:13.869431 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:13.869393 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5cf56cb474-gx24x"] Oct 29 22:43:15.478555 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:15.478523 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="877326a8-91a3-499f-8b37-4f80dfb00352" path="/var/lib/kubelet/pods/877326a8-91a3-499f-8b37-4f80dfb00352/volumes" Oct 29 22:43:18.038346 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:18.038263 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d8bc9f8cf-db28b"] Oct 29 22:43:33.910395 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:43:33.910312 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:43:36.790246 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:36.790210 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" podUID="884370ff-c54f-4d6b-af78-5c0813bb48eb" containerName="registry" containerID="cri-o://d8ea0a970d4858bf67bb65495c9ec096d88995543a29e168bac67e26b4ec4315" gracePeriod=30 Oct 29 22:43:37.078112 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.078084 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:43:37.171469 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.171430 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls\") pod \"884370ff-c54f-4d6b-af78-5c0813bb48eb\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " Oct 29 22:43:37.171666 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.171521 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-kube-api-access-t5l7q\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-kube-api-access-t5l7q\") pod \"884370ff-c54f-4d6b-af78-5c0813bb48eb\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " Oct 29 22:43:37.171666 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.171577 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884370ff-c54f-4d6b-af78-5c0813bb48eb-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/884370ff-c54f-4d6b-af78-5c0813bb48eb-ca-trust-extracted\") pod \"884370ff-c54f-4d6b-af78-5c0813bb48eb\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " Oct 29 22:43:37.171666 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.171658 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-bound-sa-token\") pod \"884370ff-c54f-4d6b-af78-5c0813bb48eb\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " Oct 29 22:43:37.171843 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.171712 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-certificates\") pod \"884370ff-c54f-4d6b-af78-5c0813bb48eb\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " Oct 29 22:43:37.171843 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.171748 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/884370ff-c54f-4d6b-af78-5c0813bb48eb-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/884370ff-c54f-4d6b-af78-5c0813bb48eb-image-registry-private-configuration\") pod \"884370ff-c54f-4d6b-af78-5c0813bb48eb\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " Oct 29 22:43:37.172607 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.172559 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-certificates" pod "884370ff-c54f-4d6b-af78-5c0813bb48eb" (UID: "884370ff-c54f-4d6b-af78-5c0813bb48eb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:37.174571 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.174527 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/884370ff-c54f-4d6b-af78-5c0813bb48eb-image-registry-private-configuration" pod "884370ff-c54f-4d6b-af78-5c0813bb48eb" (UID: "884370ff-c54f-4d6b-af78-5c0813bb48eb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:37.175092 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.175032 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls" pod "884370ff-c54f-4d6b-af78-5c0813bb48eb" (UID: "884370ff-c54f-4d6b-af78-5c0813bb48eb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:37.175235 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.175219 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-bound-sa-token" pod "884370ff-c54f-4d6b-af78-5c0813bb48eb" (UID: "884370ff-c54f-4d6b-af78-5c0813bb48eb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:37.175355 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.175330 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-kube-api-access-t5l7q" pod "884370ff-c54f-4d6b-af78-5c0813bb48eb" (UID: "884370ff-c54f-4d6b-af78-5c0813bb48eb"). InnerVolumeSpecName "kube-api-access-t5l7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:37.181342 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.181316 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884370ff-c54f-4d6b-af78-5c0813bb48eb-ca-trust-extracted" pod "884370ff-c54f-4d6b-af78-5c0813bb48eb" (UID: "884370ff-c54f-4d6b-af78-5c0813bb48eb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:37.272630 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.272591 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884370ff-c54f-4d6b-af78-5c0813bb48eb-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/884370ff-c54f-4d6b-af78-5c0813bb48eb-trusted-ca\") pod \"884370ff-c54f-4d6b-af78-5c0813bb48eb\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " Oct 29 22:43:37.272792 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.272637 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/884370ff-c54f-4d6b-af78-5c0813bb48eb-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/884370ff-c54f-4d6b-af78-5c0813bb48eb-installation-pull-secrets\") pod \"884370ff-c54f-4d6b-af78-5c0813bb48eb\" (UID: \"884370ff-c54f-4d6b-af78-5c0813bb48eb\") " Oct 29 22:43:37.272792 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.272780 5057 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:37.272792 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.272791 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5l7q\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-kube-api-access-t5l7q\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:37.272888 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.272800 5057 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/884370ff-c54f-4d6b-af78-5c0813bb48eb-ca-trust-extracted\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:37.272888 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.272809 5057 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/884370ff-c54f-4d6b-af78-5c0813bb48eb-bound-sa-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:37.272888 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.272819 5057 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/884370ff-c54f-4d6b-af78-5c0813bb48eb-registry-certificates\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:37.272888 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.272827 5057 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/884370ff-c54f-4d6b-af78-5c0813bb48eb-image-registry-private-configuration\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:37.273342 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.272952 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884370ff-c54f-4d6b-af78-5c0813bb48eb-trusted-ca" pod "884370ff-c54f-4d6b-af78-5c0813bb48eb" (UID: "884370ff-c54f-4d6b-af78-5c0813bb48eb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:37.274781 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.274761 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/884370ff-c54f-4d6b-af78-5c0813bb48eb-installation-pull-secrets" pod "884370ff-c54f-4d6b-af78-5c0813bb48eb" (UID: "884370ff-c54f-4d6b-af78-5c0813bb48eb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:37.373639 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.373558 5057 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/884370ff-c54f-4d6b-af78-5c0813bb48eb-trusted-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:37.373639 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.373588 5057 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/884370ff-c54f-4d6b-af78-5c0813bb48eb-installation-pull-secrets\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:37.893961 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.893921 5057 generic.go:356] "Generic (PLEG): container finished" podID="884370ff-c54f-4d6b-af78-5c0813bb48eb" containerID="d8ea0a970d4858bf67bb65495c9ec096d88995543a29e168bac67e26b4ec4315" exitCode=0 Oct 29 22:43:37.894392 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.893981 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" Oct 29 22:43:37.894392 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.894009 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" event={"ID":"884370ff-c54f-4d6b-af78-5c0813bb48eb","Type":"ContainerDied","Data":"d8ea0a970d4858bf67bb65495c9ec096d88995543a29e168bac67e26b4ec4315"} Oct 29 22:43:37.894392 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.894061 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78f4dff4c9-87jff" event={"ID":"884370ff-c54f-4d6b-af78-5c0813bb48eb","Type":"ContainerDied","Data":"bebb3cbd9c055a012531957e405932ac1d63d38e0a667a1e9479ae336de49d9b"} Oct 29 22:43:37.894392 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.894081 5057 scope.go:122] "RemoveContainer" containerID="d8ea0a970d4858bf67bb65495c9ec096d88995543a29e168bac67e26b4ec4315" Oct 29 22:43:37.902597 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.902579 5057 scope.go:122] "RemoveContainer" containerID="d8ea0a970d4858bf67bb65495c9ec096d88995543a29e168bac67e26b4ec4315" Oct 29 22:43:37.902900 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:43:37.902880 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ea0a970d4858bf67bb65495c9ec096d88995543a29e168bac67e26b4ec4315\": container with ID starting with d8ea0a970d4858bf67bb65495c9ec096d88995543a29e168bac67e26b4ec4315 not found: ID does not exist" containerID="d8ea0a970d4858bf67bb65495c9ec096d88995543a29e168bac67e26b4ec4315" Oct 29 22:43:37.902960 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.902908 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ea0a970d4858bf67bb65495c9ec096d88995543a29e168bac67e26b4ec4315"} err="failed to get container status \"d8ea0a970d4858bf67bb65495c9ec096d88995543a29e168bac67e26b4ec4315\": rpc error: code = NotFound desc = could not find container \"d8ea0a970d4858bf67bb65495c9ec096d88995543a29e168bac67e26b4ec4315\": container with ID starting with d8ea0a970d4858bf67bb65495c9ec096d88995543a29e168bac67e26b4ec4315 not found: ID does not exist" Oct 29 22:43:37.958930 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.958896 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78f4dff4c9-87jff"] Oct 29 22:43:37.971499 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:37.971471 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-78f4dff4c9-87jff"] Oct 29 22:43:39.478690 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:39.478656 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884370ff-c54f-4d6b-af78-5c0813bb48eb" path="/var/lib/kubelet/pods/884370ff-c54f-4d6b-af78-5c0813bb48eb/volumes" Oct 29 22:43:42.917362 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:42.917328 5057 generic.go:356] "Generic (PLEG): container finished" podID="aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8" containerID="eeb8ffb67da2a37b6680bb72557c01fd54a05a5a566c9f8a1c75f2786c0dd12c" exitCode=0 Oct 29 22:43:42.917840 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:42.917406 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-9qhv9" event={"ID":"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8","Type":"ContainerDied","Data":"eeb8ffb67da2a37b6680bb72557c01fd54a05a5a566c9f8a1c75f2786c0dd12c"} Oct 29 22:43:42.917840 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:42.917775 5057 scope.go:122] "RemoveContainer" containerID="eeb8ffb67da2a37b6680bb72557c01fd54a05a5a566c9f8a1c75f2786c0dd12c" Oct 29 22:43:43.059217 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.059163 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7d8bc9f8cf-db28b" podUID="e4841ad9-20fe-48cf-83d7-a6b95d4992ed" containerName="console" containerID="cri-o://4b8d2b20b37327ec33c095ca826bf92febc27e1c12ea51fb2547e50e87b94d24" gracePeriod=15 Oct 29 22:43:43.341942 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.341909 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:43:43.411726 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.411692 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-kube-api-access-67vrn\" (UniqueName: \"kubernetes.io/projected/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-kube-api-access-67vrn\") pod \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " Oct 29 22:43:43.411912 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.411757 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-tmp\") pod \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " Oct 29 22:43:43.411912 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.411815 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-service-ca\") pod \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " Oct 29 22:43:43.411912 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.411855 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-oauth-serving-cert\") pod \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " Oct 29 22:43:43.411912 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.411908 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-oauth-config\") pod \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " Oct 29 22:43:43.412143 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.411956 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-config\" (UniqueName: \"kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-config\") pod \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " Oct 29 22:43:43.412143 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.411988 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-serving-cert\") pod \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\" (UID: \"e4841ad9-20fe-48cf-83d7-a6b95d4992ed\") " Oct 29 22:43:43.412513 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.412387 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-oauth-serving-cert" pod "e4841ad9-20fe-48cf-83d7-a6b95d4992ed" (UID: "e4841ad9-20fe-48cf-83d7-a6b95d4992ed"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:43.412513 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.412509 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-service-ca" pod "e4841ad9-20fe-48cf-83d7-a6b95d4992ed" (UID: "e4841ad9-20fe-48cf-83d7-a6b95d4992ed"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:43.412673 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.412530 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-config" pod "e4841ad9-20fe-48cf-83d7-a6b95d4992ed" (UID: "e4841ad9-20fe-48cf-83d7-a6b95d4992ed"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:43.412673 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.412564 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-tmp" pod "e4841ad9-20fe-48cf-83d7-a6b95d4992ed" (UID: "e4841ad9-20fe-48cf-83d7-a6b95d4992ed"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:43.414556 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.414529 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-kube-api-access-67vrn" pod "e4841ad9-20fe-48cf-83d7-a6b95d4992ed" (UID: "e4841ad9-20fe-48cf-83d7-a6b95d4992ed"). InnerVolumeSpecName "kube-api-access-67vrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:43.414691 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.414658 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-oauth-config" pod "e4841ad9-20fe-48cf-83d7-a6b95d4992ed" (UID: "e4841ad9-20fe-48cf-83d7-a6b95d4992ed"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:43.415779 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.415758 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-serving-cert" pod "e4841ad9-20fe-48cf-83d7-a6b95d4992ed" (UID: "e4841ad9-20fe-48cf-83d7-a6b95d4992ed"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:43.478244 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.478150 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-vqgrg-xtvww"] Oct 29 22:43:43.478383 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.478326 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-vqgrg-xtvww" podUID="c29d1f02-5697-4d94-8a21-0a39e8f473ed" containerName="insights-gathering" containerID="cri-o://6e1c2ee96f549cfeeb4d76c428b0c5bee69900b252535e6874b48c165053a95e" gracePeriod=30 Oct 29 22:43:43.513140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.513109 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67vrn\" (UniqueName: \"kubernetes.io/projected/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-kube-api-access-67vrn\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:43.513140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.513140 5057 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-tmp\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:43.513352 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.513157 5057 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:43.513352 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.513173 5057 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-oauth-serving-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:43.513352 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.513199 5057 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-oauth-config\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:43.513352 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.513214 5057 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-config\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:43.513352 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.513228 5057 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4841ad9-20fe-48cf-83d7-a6b95d4992ed-console-serving-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:43.664369 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.664337 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-ws2qk-b8rfn"] Oct 29 22:43:43.664685 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.664672 5057 cpu_manager.go:414] "containerMap: removing container" podUID="884370ff-c54f-4d6b-af78-5c0813bb48eb" containerName="registry" Oct 29 22:43:43.664733 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.664686 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884370ff-c54f-4d6b-af78-5c0813bb48eb" containerName="registry" Oct 29 22:43:43.664733 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.664695 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e4841ad9-20fe-48cf-83d7-a6b95d4992ed" containerName="console" Oct 29 22:43:43.664733 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.664700 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4841ad9-20fe-48cf-83d7-a6b95d4992ed" containerName="console" Oct 29 22:43:43.664733 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.664712 5057 cpu_manager.go:414] "containerMap: removing container" podUID="877326a8-91a3-499f-8b37-4f80dfb00352" containerName="registry" Oct 29 22:43:43.664733 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.664716 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="877326a8-91a3-499f-8b37-4f80dfb00352" containerName="registry" Oct 29 22:43:43.664874 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.664792 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="884370ff-c54f-4d6b-af78-5c0813bb48eb" containerName="registry" Oct 29 22:43:43.664874 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.664800 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4841ad9-20fe-48cf-83d7-a6b95d4992ed" containerName="console" Oct 29 22:43:43.664874 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.664808 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="877326a8-91a3-499f-8b37-4f80dfb00352" containerName="registry" Oct 29 22:43:43.686669 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.686632 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" Oct 29 22:43:43.687864 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.687840 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ws2qk-b8rfn"] Oct 29 22:43:43.714157 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.714127 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7f8be095-10b4-46f0-8b05-d2a18c71fe73-archives-path\") pod \"periodic-gathering-ws2qk-b8rfn\" (UID: \"7f8be095-10b4-46f0-8b05-d2a18c71fe73\") " pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" Oct 29 22:43:43.714308 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.714197 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8be095-10b4-46f0-8b05-d2a18c71fe73-trusted-ca-bundle\") pod \"periodic-gathering-ws2qk-b8rfn\" (UID: \"7f8be095-10b4-46f0-8b05-d2a18c71fe73\") " pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" Oct 29 22:43:43.714362 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.714302 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8be095-10b4-46f0-8b05-d2a18c71fe73-service-ca-bundle\") pod \"periodic-gathering-ws2qk-b8rfn\" (UID: \"7f8be095-10b4-46f0-8b05-d2a18c71fe73\") " pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" Oct 29 22:43:43.714404 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.714363 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85gt5\" (UniqueName: \"kubernetes.io/projected/7f8be095-10b4-46f0-8b05-d2a18c71fe73-kube-api-access-85gt5\") pod \"periodic-gathering-ws2qk-b8rfn\" (UID: \"7f8be095-10b4-46f0-8b05-d2a18c71fe73\") " pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" Oct 29 22:43:43.768894 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.768867 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:43:43.815195 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.815157 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8be095-10b4-46f0-8b05-d2a18c71fe73-trusted-ca-bundle\") pod \"periodic-gathering-ws2qk-b8rfn\" (UID: \"7f8be095-10b4-46f0-8b05-d2a18c71fe73\") " pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" Oct 29 22:43:43.815332 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.815217 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8be095-10b4-46f0-8b05-d2a18c71fe73-service-ca-bundle\") pod \"periodic-gathering-ws2qk-b8rfn\" (UID: \"7f8be095-10b4-46f0-8b05-d2a18c71fe73\") " pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" Oct 29 22:43:43.815332 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.815254 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85gt5\" (UniqueName: \"kubernetes.io/projected/7f8be095-10b4-46f0-8b05-d2a18c71fe73-kube-api-access-85gt5\") pod \"periodic-gathering-ws2qk-b8rfn\" (UID: \"7f8be095-10b4-46f0-8b05-d2a18c71fe73\") " pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" Oct 29 22:43:43.815332 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.815284 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7f8be095-10b4-46f0-8b05-d2a18c71fe73-archives-path\") pod \"periodic-gathering-ws2qk-b8rfn\" (UID: \"7f8be095-10b4-46f0-8b05-d2a18c71fe73\") " pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" Oct 29 22:43:43.815698 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.815678 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7f8be095-10b4-46f0-8b05-d2a18c71fe73-archives-path\") pod \"periodic-gathering-ws2qk-b8rfn\" (UID: \"7f8be095-10b4-46f0-8b05-d2a18c71fe73\") " pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" Oct 29 22:43:43.815941 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.815924 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8be095-10b4-46f0-8b05-d2a18c71fe73-service-ca-bundle\") pod \"periodic-gathering-ws2qk-b8rfn\" (UID: \"7f8be095-10b4-46f0-8b05-d2a18c71fe73\") " pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" Oct 29 22:43:43.816277 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.816256 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8be095-10b4-46f0-8b05-d2a18c71fe73-trusted-ca-bundle\") pod \"periodic-gathering-ws2qk-b8rfn\" (UID: \"7f8be095-10b4-46f0-8b05-d2a18c71fe73\") " pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" Oct 29 22:43:43.840362 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.840337 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85gt5\" (UniqueName: \"kubernetes.io/projected/7f8be095-10b4-46f0-8b05-d2a18c71fe73-kube-api-access-85gt5\") pod \"periodic-gathering-ws2qk-b8rfn\" (UID: \"7f8be095-10b4-46f0-8b05-d2a18c71fe73\") " pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" Oct 29 22:43:43.916018 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.915982 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c29d1f02-5697-4d94-8a21-0a39e8f473ed-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29d1f02-5697-4d94-8a21-0a39e8f473ed-trusted-ca-bundle\") pod \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " Oct 29 22:43:43.916221 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.916043 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29d1f02-5697-4d94-8a21-0a39e8f473ed-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c29d1f02-5697-4d94-8a21-0a39e8f473ed-archives-path\") pod \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " Oct 29 22:43:43.916221 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.916092 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c29d1f02-5697-4d94-8a21-0a39e8f473ed-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29d1f02-5697-4d94-8a21-0a39e8f473ed-service-ca-bundle\") pod \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " Oct 29 22:43:43.916221 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.916113 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29d1f02-5697-4d94-8a21-0a39e8f473ed-kube-api-access-cx7m8\" (UniqueName: \"kubernetes.io/projected/c29d1f02-5697-4d94-8a21-0a39e8f473ed-kube-api-access-cx7m8\") pod \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\" (UID: \"c29d1f02-5697-4d94-8a21-0a39e8f473ed\") " Oct 29 22:43:43.916421 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.916392 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29d1f02-5697-4d94-8a21-0a39e8f473ed-archives-path" pod "c29d1f02-5697-4d94-8a21-0a39e8f473ed" (UID: "c29d1f02-5697-4d94-8a21-0a39e8f473ed"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:43.916485 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.916434 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29d1f02-5697-4d94-8a21-0a39e8f473ed-service-ca-bundle" pod "c29d1f02-5697-4d94-8a21-0a39e8f473ed" (UID: "c29d1f02-5697-4d94-8a21-0a39e8f473ed"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:43.916485 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.916455 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29d1f02-5697-4d94-8a21-0a39e8f473ed-trusted-ca-bundle" pod "c29d1f02-5697-4d94-8a21-0a39e8f473ed" (UID: "c29d1f02-5697-4d94-8a21-0a39e8f473ed"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:43.918478 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.918456 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29d1f02-5697-4d94-8a21-0a39e8f473ed-kube-api-access-cx7m8" pod "c29d1f02-5697-4d94-8a21-0a39e8f473ed" (UID: "c29d1f02-5697-4d94-8a21-0a39e8f473ed"). InnerVolumeSpecName "kube-api-access-cx7m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:43.922913 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.922886 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-9qhv9" event={"ID":"aa5c9f0f-7f65-4cbc-a95a-a92e52c18aa8","Type":"ContainerStarted","Data":"f5aaf0fcd19bee513630e83c9402b70c4e4bd5953ca950cb557cfe68dce8356f"} Oct 29 22:43:43.924037 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.924018 5057 generic.go:356] "Generic (PLEG): container finished" podID="e4841ad9-20fe-48cf-83d7-a6b95d4992ed" containerID="4b8d2b20b37327ec33c095ca826bf92febc27e1c12ea51fb2547e50e87b94d24" exitCode=0 Oct 29 22:43:43.924136 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.924056 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d8bc9f8cf-db28b" Oct 29 22:43:43.924136 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.924104 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d8bc9f8cf-db28b" event={"ID":"e4841ad9-20fe-48cf-83d7-a6b95d4992ed","Type":"ContainerDied","Data":"4b8d2b20b37327ec33c095ca826bf92febc27e1c12ea51fb2547e50e87b94d24"} Oct 29 22:43:43.924261 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.924144 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d8bc9f8cf-db28b" event={"ID":"e4841ad9-20fe-48cf-83d7-a6b95d4992ed","Type":"ContainerDied","Data":"b40b0bc67d568263f18df0f1cb74b98517308dbe742d9410e3fd173b3559100b"} Oct 29 22:43:43.924261 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.924165 5057 scope.go:122] "RemoveContainer" containerID="4b8d2b20b37327ec33c095ca826bf92febc27e1c12ea51fb2547e50e87b94d24" Oct 29 22:43:43.925392 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.925369 5057 generic.go:356] "Generic (PLEG): container finished" podID="c29d1f02-5697-4d94-8a21-0a39e8f473ed" containerID="6e1c2ee96f549cfeeb4d76c428b0c5bee69900b252535e6874b48c165053a95e" exitCode=2 Oct 29 22:43:43.925491 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.925402 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vqgrg-xtvww" event={"ID":"c29d1f02-5697-4d94-8a21-0a39e8f473ed","Type":"ContainerDied","Data":"6e1c2ee96f549cfeeb4d76c428b0c5bee69900b252535e6874b48c165053a95e"} Oct 29 22:43:43.925491 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.925425 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vqgrg-xtvww" Oct 29 22:43:43.925491 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.925449 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vqgrg-xtvww" event={"ID":"c29d1f02-5697-4d94-8a21-0a39e8f473ed","Type":"ContainerDied","Data":"9f59a804858118aa038ff14cb50db8dcd52f6a3704bb85fbcf18447550d056fe"} Oct 29 22:43:43.935119 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.935102 5057 scope.go:122] "RemoveContainer" containerID="4b8d2b20b37327ec33c095ca826bf92febc27e1c12ea51fb2547e50e87b94d24" Oct 29 22:43:43.935657 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:43:43.935631 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b8d2b20b37327ec33c095ca826bf92febc27e1c12ea51fb2547e50e87b94d24\": container with ID starting with 4b8d2b20b37327ec33c095ca826bf92febc27e1c12ea51fb2547e50e87b94d24 not found: ID does not exist" containerID="4b8d2b20b37327ec33c095ca826bf92febc27e1c12ea51fb2547e50e87b94d24" Oct 29 22:43:43.935736 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.935662 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b8d2b20b37327ec33c095ca826bf92febc27e1c12ea51fb2547e50e87b94d24"} err="failed to get container status \"4b8d2b20b37327ec33c095ca826bf92febc27e1c12ea51fb2547e50e87b94d24\": rpc error: code = NotFound desc = could not find container \"4b8d2b20b37327ec33c095ca826bf92febc27e1c12ea51fb2547e50e87b94d24\": container with ID starting with 4b8d2b20b37327ec33c095ca826bf92febc27e1c12ea51fb2547e50e87b94d24 not found: ID does not exist" Oct 29 22:43:43.935736 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.935678 5057 scope.go:122] "RemoveContainer" containerID="6e1c2ee96f549cfeeb4d76c428b0c5bee69900b252535e6874b48c165053a95e" Oct 29 22:43:43.946244 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.946223 5057 scope.go:122] "RemoveContainer" containerID="6e1c2ee96f549cfeeb4d76c428b0c5bee69900b252535e6874b48c165053a95e" Oct 29 22:43:43.946515 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:43:43.946492 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e1c2ee96f549cfeeb4d76c428b0c5bee69900b252535e6874b48c165053a95e\": container with ID starting with 6e1c2ee96f549cfeeb4d76c428b0c5bee69900b252535e6874b48c165053a95e not found: ID does not exist" containerID="6e1c2ee96f549cfeeb4d76c428b0c5bee69900b252535e6874b48c165053a95e" Oct 29 22:43:43.946594 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:43.946520 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e1c2ee96f549cfeeb4d76c428b0c5bee69900b252535e6874b48c165053a95e"} err="failed to get container status \"6e1c2ee96f549cfeeb4d76c428b0c5bee69900b252535e6874b48c165053a95e\": rpc error: code = NotFound desc = could not find container \"6e1c2ee96f549cfeeb4d76c428b0c5bee69900b252535e6874b48c165053a95e\": container with ID starting with 6e1c2ee96f549cfeeb4d76c428b0c5bee69900b252535e6874b48c165053a95e not found: ID does not exist" Oct 29 22:43:44.004953 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:44.004873 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d8bc9f8cf-db28b"] Oct 29 22:43:44.008923 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:44.008900 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d8bc9f8cf-db28b"] Oct 29 22:43:44.017495 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:44.017465 5057 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29d1f02-5697-4d94-8a21-0a39e8f473ed-trusted-ca-bundle\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:44.017495 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:44.017497 5057 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c29d1f02-5697-4d94-8a21-0a39e8f473ed-archives-path\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:44.017726 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:44.017514 5057 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29d1f02-5697-4d94-8a21-0a39e8f473ed-service-ca-bundle\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:44.017726 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:44.017534 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx7m8\" (UniqueName: \"kubernetes.io/projected/c29d1f02-5697-4d94-8a21-0a39e8f473ed-kube-api-access-cx7m8\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:43:44.020217 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:44.020197 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" Oct 29 22:43:44.055602 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:44.055575 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-vqgrg-xtvww"] Oct 29 22:43:44.059799 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:44.059771 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-vqgrg-xtvww"] Oct 29 22:43:44.324437 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:44.324363 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ws2qk-b8rfn"] Oct 29 22:43:44.327259 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:43:44.327226 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8be095_10b4_46f0_8b05_d2a18c71fe73.slice/crio-a254598486d6164142da578337d7b49ba007100a01385fd771e0316cc3bd95c0 WatchSource:0}: Error finding container a254598486d6164142da578337d7b49ba007100a01385fd771e0316cc3bd95c0: Status 404 returned error can't find the container with id a254598486d6164142da578337d7b49ba007100a01385fd771e0316cc3bd95c0 Oct 29 22:43:44.931068 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:44.931031 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" event={"ID":"7f8be095-10b4-46f0-8b05-d2a18c71fe73","Type":"ContainerStarted","Data":"c89bb1ace8952c44f3b0de90bc4aa4a530fbb9f297cc767772206add5bc50db4"} Oct 29 22:43:44.931512 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:44.931083 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" event={"ID":"7f8be095-10b4-46f0-8b05-d2a18c71fe73","Type":"ContainerStarted","Data":"a254598486d6164142da578337d7b49ba007100a01385fd771e0316cc3bd95c0"} Oct 29 22:43:44.971984 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:44.971934 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" podStartSLOduration=1.971921306 podStartE2EDuration="1.971921306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:44.971132073 +0000 UTC m=+71.622630605" watchObservedRunningTime="2026-10-29 22:43:44.971921306 +0000 UTC m=+71.623419832" Oct 29 22:43:45.477988 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:45.477954 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c29d1f02-5697-4d94-8a21-0a39e8f473ed" path="/var/lib/kubelet/pods/c29d1f02-5697-4d94-8a21-0a39e8f473ed/volumes" Oct 29 22:43:45.478434 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:43:45.478418 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4841ad9-20fe-48cf-83d7-a6b95d4992ed" path="/var/lib/kubelet/pods/e4841ad9-20fe-48cf-83d7-a6b95d4992ed/volumes" Oct 29 22:44:03.913835 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:44:03.913794 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:44:33.912785 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:44:33.912701 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:45:03.911581 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:45:03.911542 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:45:08.262081 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:08.262046 5057 generic.go:356] "Generic (PLEG): container finished" podID="7f8be095-10b4-46f0-8b05-d2a18c71fe73" containerID="c89bb1ace8952c44f3b0de90bc4aa4a530fbb9f297cc767772206add5bc50db4" exitCode=0 Oct 29 22:45:08.262478 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:08.262121 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" event={"ID":"7f8be095-10b4-46f0-8b05-d2a18c71fe73","Type":"ContainerDied","Data":"c89bb1ace8952c44f3b0de90bc4aa4a530fbb9f297cc767772206add5bc50db4"} Oct 29 22:45:09.397925 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:09.397901 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" Oct 29 22:45:09.517614 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:09.517546 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f8be095-10b4-46f0-8b05-d2a18c71fe73-kube-api-access-85gt5\" (UniqueName: \"kubernetes.io/projected/7f8be095-10b4-46f0-8b05-d2a18c71fe73-kube-api-access-85gt5\") pod \"7f8be095-10b4-46f0-8b05-d2a18c71fe73\" (UID: \"7f8be095-10b4-46f0-8b05-d2a18c71fe73\") " Oct 29 22:45:09.517614 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:09.517597 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f8be095-10b4-46f0-8b05-d2a18c71fe73-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8be095-10b4-46f0-8b05-d2a18c71fe73-service-ca-bundle\") pod \"7f8be095-10b4-46f0-8b05-d2a18c71fe73\" (UID: \"7f8be095-10b4-46f0-8b05-d2a18c71fe73\") " Oct 29 22:45:09.517819 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:09.517695 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f8be095-10b4-46f0-8b05-d2a18c71fe73-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8be095-10b4-46f0-8b05-d2a18c71fe73-trusted-ca-bundle\") pod \"7f8be095-10b4-46f0-8b05-d2a18c71fe73\" (UID: \"7f8be095-10b4-46f0-8b05-d2a18c71fe73\") " Oct 29 22:45:09.517819 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:09.517722 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8be095-10b4-46f0-8b05-d2a18c71fe73-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7f8be095-10b4-46f0-8b05-d2a18c71fe73-archives-path\") pod \"7f8be095-10b4-46f0-8b05-d2a18c71fe73\" (UID: \"7f8be095-10b4-46f0-8b05-d2a18c71fe73\") " Oct 29 22:45:09.518038 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:09.518004 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f8be095-10b4-46f0-8b05-d2a18c71fe73-service-ca-bundle" pod "7f8be095-10b4-46f0-8b05-d2a18c71fe73" (UID: "7f8be095-10b4-46f0-8b05-d2a18c71fe73"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:09.518038 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:09.518024 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f8be095-10b4-46f0-8b05-d2a18c71fe73-trusted-ca-bundle" pod "7f8be095-10b4-46f0-8b05-d2a18c71fe73" (UID: "7f8be095-10b4-46f0-8b05-d2a18c71fe73"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:09.518217 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:09.518082 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8be095-10b4-46f0-8b05-d2a18c71fe73-archives-path" pod "7f8be095-10b4-46f0-8b05-d2a18c71fe73" (UID: "7f8be095-10b4-46f0-8b05-d2a18c71fe73"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:09.519720 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:09.519702 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8be095-10b4-46f0-8b05-d2a18c71fe73-kube-api-access-85gt5" pod "7f8be095-10b4-46f0-8b05-d2a18c71fe73" (UID: "7f8be095-10b4-46f0-8b05-d2a18c71fe73"). InnerVolumeSpecName "kube-api-access-85gt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:09.618458 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:09.618409 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85gt5\" (UniqueName: \"kubernetes.io/projected/7f8be095-10b4-46f0-8b05-d2a18c71fe73-kube-api-access-85gt5\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:45:09.618458 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:09.618452 5057 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8be095-10b4-46f0-8b05-d2a18c71fe73-service-ca-bundle\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:45:09.618458 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:09.618462 5057 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8be095-10b4-46f0-8b05-d2a18c71fe73-trusted-ca-bundle\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:45:09.618458 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:09.618472 5057 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7f8be095-10b4-46f0-8b05-d2a18c71fe73-archives-path\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:45:10.271568 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:10.271538 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" Oct 29 22:45:10.271568 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:10.271563 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ws2qk-b8rfn" event={"ID":"7f8be095-10b4-46f0-8b05-d2a18c71fe73","Type":"ContainerDied","Data":"a254598486d6164142da578337d7b49ba007100a01385fd771e0316cc3bd95c0"} Oct 29 22:45:10.271819 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:10.271605 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a254598486d6164142da578337d7b49ba007100a01385fd771e0316cc3bd95c0" Oct 29 22:45:33.911734 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:45:33.911694 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:45:47.827735 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.827700 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf"] Oct 29 22:45:47.828221 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.828023 5057 cpu_manager.go:414] "containerMap: removing container" podUID="c29d1f02-5697-4d94-8a21-0a39e8f473ed" containerName="insights-gathering" Oct 29 22:45:47.828221 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.828034 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c29d1f02-5697-4d94-8a21-0a39e8f473ed" containerName="insights-gathering" Oct 29 22:45:47.828221 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.828050 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7f8be095-10b4-46f0-8b05-d2a18c71fe73" containerName="insights-gathering" Oct 29 22:45:47.828221 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.828055 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f8be095-10b4-46f0-8b05-d2a18c71fe73" containerName="insights-gathering" Oct 29 22:45:47.828221 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.828111 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="c29d1f02-5697-4d94-8a21-0a39e8f473ed" containerName="insights-gathering" Oct 29 22:45:47.828221 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.828120 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f8be095-10b4-46f0-8b05-d2a18c71fe73" containerName="insights-gathering" Oct 29 22:45:47.829897 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.829878 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.831812 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.831788 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdn2z\" (UniqueName: \"kubernetes.io/projected/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-kube-api-access-rdn2z\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.831937 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.831828 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.831937 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.831855 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.832051 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.831961 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.832051 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.832003 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.832149 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.832124 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.832212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.832170 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.832262 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.832215 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.832262 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.832236 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.847164 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.847140 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-4lkhj\"" Oct 29 22:45:47.859685 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.859661 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 29 22:45:47.861427 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.861405 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf"] Oct 29 22:45:47.932586 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.932552 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.932586 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.932588 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.932827 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.932608 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.932827 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.932624 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.932827 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.932754 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdn2z\" (UniqueName: \"kubernetes.io/projected/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-kube-api-access-rdn2z\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.932827 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.932811 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.933036 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.932843 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.933036 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.932889 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.933036 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.932935 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.933036 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.933006 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.933229 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.933149 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.933229 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.933167 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.933341 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.933317 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.933426 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.933407 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.935357 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.935326 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.935479 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.935420 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.978328 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.978302 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:47.978941 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:47.978922 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdn2z\" (UniqueName: \"kubernetes.io/projected/12f9734d-d6d4-4a4e-a0b1-13ebd9505d54-kube-api-access-rdn2z\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf\" (UID: \"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:48.141475 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:48.141386 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:48.449677 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:48.449540 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf"] Oct 29 22:45:48.452305 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:45:48.452276 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f9734d_d6d4_4a4e_a0b1_13ebd9505d54.slice/crio-c1c2f0b70bd455e2e92fa20cb179fec9d55a4959ab807136e1f46f3553b4f9d5 WatchSource:0}: Error finding container c1c2f0b70bd455e2e92fa20cb179fec9d55a4959ab807136e1f46f3553b4f9d5: Status 404 returned error can't find the container with id c1c2f0b70bd455e2e92fa20cb179fec9d55a4959ab807136e1f46f3553b4f9d5 Oct 29 22:45:49.421919 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:49.421875 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" event={"ID":"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54","Type":"ContainerStarted","Data":"c1c2f0b70bd455e2e92fa20cb179fec9d55a4959ab807136e1f46f3553b4f9d5"} Oct 29 22:45:50.996060 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:50.996028 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:45:50.996330 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:50.996096 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:45:50.996330 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:50.996128 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:45:51.438295 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:51.438256 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" event={"ID":"12f9734d-d6d4-4a4e-a0b1-13ebd9505d54","Type":"ContainerStarted","Data":"ccbeaf20f41bbf8e8d46d5e99a82a11876af0e7bd382c378740429a7f2af015f"} Oct 29 22:45:51.496879 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:51.496827 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" podStartSLOduration=1.9557677679999999 podStartE2EDuration="4.496814664s" totalImagesPullingTime="2.541046896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:51.496537503 +0000 UTC m=+198.148036033" watchObservedRunningTime="2026-10-29 22:45:51.496814664 +0000 UTC m=+198.148313193" Oct 29 22:45:52.141614 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:52.141576 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:52.146659 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:52.146634 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:52.442528 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:52.442501 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:45:52.443617 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:45:52.443597 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf" Oct 29 22:46:03.912414 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:46:03.912374 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:46:28.517164 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.517110 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg"] Oct 29 22:46:28.524614 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.524592 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:28.564103 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.564074 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 29 22:46:28.566739 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.566717 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-c59b2\"" Oct 29 22:46:28.576719 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.576701 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 29 22:46:28.576719 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.576710 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 29 22:46:28.577108 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.577087 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 29 22:46:28.594239 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.594219 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg"] Oct 29 22:46:28.656484 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.656455 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 29 22:46:28.667811 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.667780 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45-webhook-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-5rdbg\" (UID: \"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:28.667938 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.667822 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45-tmp\") pod \"opendatahub-operator-controller-manager-844ffc67f7-5rdbg\" (UID: \"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:28.667938 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.667906 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45-manifests\") pod \"opendatahub-operator-controller-manager-844ffc67f7-5rdbg\" (UID: \"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:28.668018 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.667939 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45-metrics-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-5rdbg\" (UID: \"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:28.668018 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.668006 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgr26\" (UniqueName: \"kubernetes.io/projected/4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45-kube-api-access-wgr26\") pod \"opendatahub-operator-controller-manager-844ffc67f7-5rdbg\" (UID: \"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:28.668082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.668040 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45-apiservice-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-5rdbg\" (UID: \"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:28.768648 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.768567 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45-apiservice-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-5rdbg\" (UID: \"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:28.768648 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.768608 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45-webhook-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-5rdbg\" (UID: \"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:28.768828 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.768726 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45-tmp\") pod \"opendatahub-operator-controller-manager-844ffc67f7-5rdbg\" (UID: \"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:28.768828 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.768774 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45-manifests\") pod \"opendatahub-operator-controller-manager-844ffc67f7-5rdbg\" (UID: \"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:28.768828 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.768813 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45-metrics-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-5rdbg\" (UID: \"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:28.768963 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.768852 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgr26\" (UniqueName: \"kubernetes.io/projected/4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45-kube-api-access-wgr26\") pod \"opendatahub-operator-controller-manager-844ffc67f7-5rdbg\" (UID: \"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:28.769245 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.769217 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45-manifests\") pod \"opendatahub-operator-controller-manager-844ffc67f7-5rdbg\" (UID: \"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:28.771175 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.771147 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45-webhook-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-5rdbg\" (UID: \"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:28.771295 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.771174 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45-apiservice-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-5rdbg\" (UID: \"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:28.771396 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.771374 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45-tmp\") pod \"opendatahub-operator-controller-manager-844ffc67f7-5rdbg\" (UID: \"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:28.771505 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.771491 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45-metrics-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-5rdbg\" (UID: \"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:28.796227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.796204 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgr26\" (UniqueName: \"kubernetes.io/projected/4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45-kube-api-access-wgr26\") pod \"opendatahub-operator-controller-manager-844ffc67f7-5rdbg\" (UID: \"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:28.841626 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:28.841593 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:29.180734 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:29.180709 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg"] Oct 29 22:46:29.183282 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:46:29.183255 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4057ebe9_4fd4_43e6_a1f0_d95bb1af9a45.slice/crio-8a17d720b9e13b8551348f2e6a3dac50052467e87ebd2a2f6b29530632588791 WatchSource:0}: Error finding container 8a17d720b9e13b8551348f2e6a3dac50052467e87ebd2a2f6b29530632588791: Status 404 returned error can't find the container with id 8a17d720b9e13b8551348f2e6a3dac50052467e87ebd2a2f6b29530632588791 Oct 29 22:46:29.583249 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:29.583150 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" event={"ID":"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45","Type":"ContainerStarted","Data":"8a17d720b9e13b8551348f2e6a3dac50052467e87ebd2a2f6b29530632588791"} Oct 29 22:46:31.182528 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:31.182485 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-fbgn7"] Oct 29 22:46:31.186359 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:31.186334 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fbgn7" Oct 29 22:46:31.229348 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:31.229282 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 29 22:46:31.229348 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:31.229309 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 29 22:46:31.229348 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:31.229315 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 29 22:46:31.229614 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:31.229404 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-ct8xd\"" Oct 29 22:46:31.230791 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:31.230762 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-fbgn7"] Oct 29 22:46:31.291225 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:31.291162 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99mgv\" (UniqueName: \"kubernetes.io/projected/08ce15dc-71d7-4b48-9ad4-75a826414293-kube-api-access-99mgv\") pod \"dns-operator-controller-manager-8557cbdb84-fbgn7\" (UID: \"08ce15dc-71d7-4b48-9ad4-75a826414293\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fbgn7" Oct 29 22:46:31.392285 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:31.392251 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99mgv\" (UniqueName: \"kubernetes.io/projected/08ce15dc-71d7-4b48-9ad4-75a826414293-kube-api-access-99mgv\") pod \"dns-operator-controller-manager-8557cbdb84-fbgn7\" (UID: \"08ce15dc-71d7-4b48-9ad4-75a826414293\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fbgn7" Oct 29 22:46:31.416630 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:31.416598 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99mgv\" (UniqueName: \"kubernetes.io/projected/08ce15dc-71d7-4b48-9ad4-75a826414293-kube-api-access-99mgv\") pod \"dns-operator-controller-manager-8557cbdb84-fbgn7\" (UID: \"08ce15dc-71d7-4b48-9ad4-75a826414293\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fbgn7" Oct 29 22:46:31.501382 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:31.501297 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fbgn7" Oct 29 22:46:31.818754 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:31.818722 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-fbgn7"] Oct 29 22:46:31.821615 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:46:31.821570 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ce15dc_71d7_4b48_9ad4_75a826414293.slice/crio-3ff8e2e124960b68106c87d7d26f29b75aa8026978cc15d628a1cc35ed7f1a9d WatchSource:0}: Error finding container 3ff8e2e124960b68106c87d7d26f29b75aa8026978cc15d628a1cc35ed7f1a9d: Status 404 returned error can't find the container with id 3ff8e2e124960b68106c87d7d26f29b75aa8026978cc15d628a1cc35ed7f1a9d Oct 29 22:46:32.609581 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:32.609542 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" event={"ID":"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45","Type":"ContainerStarted","Data":"d60da473003fea9749a8e69bdd40ef9aa79d6d1f29df3a741363a35ac3bb7f1c"} Oct 29 22:46:32.611379 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:32.611347 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fbgn7" event={"ID":"08ce15dc-71d7-4b48-9ad4-75a826414293","Type":"ContainerStarted","Data":"3ff8e2e124960b68106c87d7d26f29b75aa8026978cc15d628a1cc35ed7f1a9d"} Oct 29 22:46:33.617717 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:33.617681 5057 generic.go:356] "Generic (PLEG): container finished" podID="4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45" containerID="d60da473003fea9749a8e69bdd40ef9aa79d6d1f29df3a741363a35ac3bb7f1c" exitCode=0 Oct 29 22:46:33.618146 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:33.617724 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" event={"ID":"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45","Type":"ContainerDied","Data":"d60da473003fea9749a8e69bdd40ef9aa79d6d1f29df3a741363a35ac3bb7f1c"} Oct 29 22:46:33.913020 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:46:33.912950 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:46:34.355486 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:34.355455 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 29 22:46:34.624391 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:34.624304 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" event={"ID":"4057ebe9-4fd4-43e6-a1f0-d95bb1af9a45","Type":"ContainerStarted","Data":"b7d3f37386fd3c30e3a48d5db113784dafd73f906aaa0e484a6923d5218c0c41"} Oct 29 22:46:34.624765 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:34.624419 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:34.626501 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:34.626470 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fbgn7" event={"ID":"08ce15dc-71d7-4b48-9ad4-75a826414293","Type":"ContainerStarted","Data":"5c7b3c3c552b4516aa9c2e4de7345e5cae230fd25a0742d3e60c0639c99d3f23"} Oct 29 22:46:34.626647 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:34.626612 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fbgn7" Oct 29 22:46:34.694387 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:34.694328 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" podStartSLOduration=2.86141189 podStartE2EDuration="6.694313875s" totalImagesPullingTime="3.317791248s" totalInitContainerRuntime="515.110737ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:34.69245784 +0000 UTC m=+241.343956382" watchObservedRunningTime="2026-10-29 22:46:34.694313875 +0000 UTC m=+241.345812402" Oct 29 22:46:34.763029 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:34.762976 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fbgn7" podStartSLOduration=1.246550362 podStartE2EDuration="3.76296116s" totalImagesPullingTime="2.516410798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:34.761661247 +0000 UTC m=+241.413159778" watchObservedRunningTime="2026-10-29 22:46:34.76296116 +0000 UTC m=+241.414459690" Oct 29 22:46:34.867147 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:34.867116 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-j2gvf"] Oct 29 22:46:34.870529 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:34.870511 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-j2gvf" Oct 29 22:46:34.886680 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:34.886640 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 29 22:46:34.886803 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:34.886733 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8kwg7\"" Oct 29 22:46:34.888404 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:34.888383 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 29 22:46:34.890941 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:34.890911 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-j2gvf"] Oct 29 22:46:35.023708 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:35.023668 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kbgf\" (UniqueName: \"kubernetes.io/projected/cabd04f5-8346-495d-9f50-6c38a692df10-kube-api-access-7kbgf\") pod \"kuadrant-console-plugin-66c4fb575d-j2gvf\" (UID: \"cabd04f5-8346-495d-9f50-6c38a692df10\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-j2gvf" Oct 29 22:46:35.023708 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:35.023710 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cabd04f5-8346-495d-9f50-6c38a692df10-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-j2gvf\" (UID: \"cabd04f5-8346-495d-9f50-6c38a692df10\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-j2gvf" Oct 29 22:46:35.023931 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:35.023848 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cabd04f5-8346-495d-9f50-6c38a692df10-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-j2gvf\" (UID: \"cabd04f5-8346-495d-9f50-6c38a692df10\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-j2gvf" Oct 29 22:46:35.125297 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:35.125261 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cabd04f5-8346-495d-9f50-6c38a692df10-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-j2gvf\" (UID: \"cabd04f5-8346-495d-9f50-6c38a692df10\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-j2gvf" Oct 29 22:46:35.125460 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:35.125317 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kbgf\" (UniqueName: \"kubernetes.io/projected/cabd04f5-8346-495d-9f50-6c38a692df10-kube-api-access-7kbgf\") pod \"kuadrant-console-plugin-66c4fb575d-j2gvf\" (UID: \"cabd04f5-8346-495d-9f50-6c38a692df10\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-j2gvf" Oct 29 22:46:35.125460 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:35.125346 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cabd04f5-8346-495d-9f50-6c38a692df10-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-j2gvf\" (UID: \"cabd04f5-8346-495d-9f50-6c38a692df10\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-j2gvf" Oct 29 22:46:35.126021 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:35.126002 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cabd04f5-8346-495d-9f50-6c38a692df10-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-j2gvf\" (UID: \"cabd04f5-8346-495d-9f50-6c38a692df10\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-j2gvf" Oct 29 22:46:35.127827 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:35.127771 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cabd04f5-8346-495d-9f50-6c38a692df10-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-j2gvf\" (UID: \"cabd04f5-8346-495d-9f50-6c38a692df10\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-j2gvf" Oct 29 22:46:35.150006 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:35.149984 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kbgf\" (UniqueName: \"kubernetes.io/projected/cabd04f5-8346-495d-9f50-6c38a692df10-kube-api-access-7kbgf\") pod \"kuadrant-console-plugin-66c4fb575d-j2gvf\" (UID: \"cabd04f5-8346-495d-9f50-6c38a692df10\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-j2gvf" Oct 29 22:46:35.193211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:35.193175 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-j2gvf" Oct 29 22:46:35.502907 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:35.502880 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-j2gvf"] Oct 29 22:46:35.504460 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:46:35.504435 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcabd04f5_8346_495d_9f50_6c38a692df10.slice/crio-551321a7dd131f1af0722fcc7380115197e7269f85453570cc51c2bcbdb68384 WatchSource:0}: Error finding container 551321a7dd131f1af0722fcc7380115197e7269f85453570cc51c2bcbdb68384: Status 404 returned error can't find the container with id 551321a7dd131f1af0722fcc7380115197e7269f85453570cc51c2bcbdb68384 Oct 29 22:46:35.631636 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:35.631597 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-j2gvf" event={"ID":"cabd04f5-8346-495d-9f50-6c38a692df10","Type":"ContainerStarted","Data":"551321a7dd131f1af0722fcc7380115197e7269f85453570cc51c2bcbdb68384"} Oct 29 22:46:40.241126 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:40.241051 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-gmpzb"] Oct 29 22:46:40.244716 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:40.244696 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-gmpzb" Oct 29 22:46:40.275432 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:40.275407 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-9pmnc\"" Oct 29 22:46:40.276888 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:40.276864 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-gmpzb"] Oct 29 22:46:40.371905 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:40.371874 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chb45\" (UniqueName: \"kubernetes.io/projected/a58ed6fc-270f-4377-83b4-486019dbe952-kube-api-access-chb45\") pod \"limitador-operator-controller-manager-5778bb4449-gmpzb\" (UID: \"a58ed6fc-270f-4377-83b4-486019dbe952\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-gmpzb" Oct 29 22:46:40.473230 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:40.473167 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chb45\" (UniqueName: \"kubernetes.io/projected/a58ed6fc-270f-4377-83b4-486019dbe952-kube-api-access-chb45\") pod \"limitador-operator-controller-manager-5778bb4449-gmpzb\" (UID: \"a58ed6fc-270f-4377-83b4-486019dbe952\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-gmpzb" Oct 29 22:46:40.497544 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:40.497478 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chb45\" (UniqueName: \"kubernetes.io/projected/a58ed6fc-270f-4377-83b4-486019dbe952-kube-api-access-chb45\") pod \"limitador-operator-controller-manager-5778bb4449-gmpzb\" (UID: \"a58ed6fc-270f-4377-83b4-486019dbe952\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-gmpzb" Oct 29 22:46:40.556920 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:40.556887 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-gmpzb" Oct 29 22:46:40.654966 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:40.654932 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-j2gvf" event={"ID":"cabd04f5-8346-495d-9f50-6c38a692df10","Type":"ContainerStarted","Data":"633023e4886912717fcb65ee569c8b961021a9cb8ea19bcfd1f3c7515df0eb07"} Oct 29 22:46:40.726556 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:40.726506 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-j2gvf" podStartSLOduration=2.289031828 podStartE2EDuration="6.726491666s" totalImagesPullingTime="4.437459838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:40.725848482 +0000 UTC m=+247.377347011" watchObservedRunningTime="2026-10-29 22:46:40.726491666 +0000 UTC m=+247.377990193" Oct 29 22:46:40.872579 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:40.872551 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-gmpzb"] Oct 29 22:46:40.873731 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:46:40.873701 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58ed6fc_270f_4377_83b4_486019dbe952.slice/crio-a47876464e820fc903d741098f2dd8f018aa60d8c06107cb9cc999696d9e0914 WatchSource:0}: Error finding container a47876464e820fc903d741098f2dd8f018aa60d8c06107cb9cc999696d9e0914: Status 404 returned error can't find the container with id a47876464e820fc903d741098f2dd8f018aa60d8c06107cb9cc999696d9e0914 Oct 29 22:46:41.660308 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:41.660275 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-gmpzb" event={"ID":"a58ed6fc-270f-4377-83b4-486019dbe952","Type":"ContainerStarted","Data":"a47876464e820fc903d741098f2dd8f018aa60d8c06107cb9cc999696d9e0914"} Oct 29 22:46:42.666403 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:42.666318 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-gmpzb" event={"ID":"a58ed6fc-270f-4377-83b4-486019dbe952","Type":"ContainerStarted","Data":"7b39960eb350d639e8330a5af76de1f7aca01b13d05ce2b360aff3a6f85a11b9"} Oct 29 22:46:42.666793 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:42.666485 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-gmpzb" Oct 29 22:46:42.710990 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:42.710932 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-gmpzb" podStartSLOduration=1.396555744 podStartE2EDuration="2.710919818s" totalImagesPullingTime="1.314364074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:42.709150421 +0000 UTC m=+249.360648951" watchObservedRunningTime="2026-10-29 22:46:42.710919818 +0000 UTC m=+249.362418347" Oct 29 22:46:43.433471 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:43.433436 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt"] Oct 29 22:46:43.436917 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:43.436901 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt" Oct 29 22:46:43.462944 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:43.462919 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt"] Oct 29 22:46:43.474026 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:43.473986 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-f8bng\"" Oct 29 22:46:43.602350 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:43.602320 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55c4bb94-26cc-45e9-ac4c-3542b1758fcb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zm2vt\" (UID: \"55c4bb94-26cc-45e9-ac4c-3542b1758fcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt" Oct 29 22:46:43.602525 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:43.602369 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhm65\" (UniqueName: \"kubernetes.io/projected/55c4bb94-26cc-45e9-ac4c-3542b1758fcb-kube-api-access-zhm65\") pod \"kuadrant-operator-controller-manager-678d87766f-zm2vt\" (UID: \"55c4bb94-26cc-45e9-ac4c-3542b1758fcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt" Oct 29 22:46:43.703622 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:43.703540 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhm65\" (UniqueName: \"kubernetes.io/projected/55c4bb94-26cc-45e9-ac4c-3542b1758fcb-kube-api-access-zhm65\") pod \"kuadrant-operator-controller-manager-678d87766f-zm2vt\" (UID: \"55c4bb94-26cc-45e9-ac4c-3542b1758fcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt" Oct 29 22:46:43.703988 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:43.703633 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55c4bb94-26cc-45e9-ac4c-3542b1758fcb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zm2vt\" (UID: \"55c4bb94-26cc-45e9-ac4c-3542b1758fcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt" Oct 29 22:46:43.704034 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:43.704013 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55c4bb94-26cc-45e9-ac4c-3542b1758fcb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zm2vt\" (UID: \"55c4bb94-26cc-45e9-ac4c-3542b1758fcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt" Oct 29 22:46:43.733827 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:43.733806 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhm65\" (UniqueName: \"kubernetes.io/projected/55c4bb94-26cc-45e9-ac4c-3542b1758fcb-kube-api-access-zhm65\") pod \"kuadrant-operator-controller-manager-678d87766f-zm2vt\" (UID: \"55c4bb94-26cc-45e9-ac4c-3542b1758fcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt" Oct 29 22:46:43.749676 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:43.749653 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt" Oct 29 22:46:44.082716 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:44.082630 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt"] Oct 29 22:46:44.095397 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:46:44.095371 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c4bb94_26cc_45e9_ac4c_3542b1758fcb.slice/crio-5ca18296cdb16e7aca510c5fde7d2859c71b230c703c5f58204bb0be8e9c78df WatchSource:0}: Error finding container 5ca18296cdb16e7aca510c5fde7d2859c71b230c703c5f58204bb0be8e9c78df: Status 404 returned error can't find the container with id 5ca18296cdb16e7aca510c5fde7d2859c71b230c703c5f58204bb0be8e9c78df Oct 29 22:46:44.676350 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:44.676314 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt" event={"ID":"55c4bb94-26cc-45e9-ac4c-3542b1758fcb","Type":"ContainerStarted","Data":"5ca18296cdb16e7aca510c5fde7d2859c71b230c703c5f58204bb0be8e9c78df"} Oct 29 22:46:45.634702 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:45.634668 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-5rdbg" Oct 29 22:46:45.635103 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:45.634758 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fbgn7" Oct 29 22:46:48.695721 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:48.695683 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt" event={"ID":"55c4bb94-26cc-45e9-ac4c-3542b1758fcb","Type":"ContainerStarted","Data":"821940f367cba577a59738e0d12b4f298326901950decec0b4654b3a0423689a"} Oct 29 22:46:48.696079 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:48.695838 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt" Oct 29 22:46:48.778168 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:48.778115 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt" podStartSLOduration=1.309598541 podStartE2EDuration="5.778100555s" totalImagesPullingTime="4.468502014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:48.773036083 +0000 UTC m=+255.424534613" watchObservedRunningTime="2026-10-29 22:46:48.778100555 +0000 UTC m=+255.429599082" Oct 29 22:46:53.672592 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:53.672562 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-gmpzb" Oct 29 22:46:59.702815 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:46:59.702783 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt" Oct 29 22:47:03.912650 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:47:03.912612 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:47:12.502158 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:12.502125 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-xf9r8"] Oct 29 22:47:12.522269 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:12.522243 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xf9r8" Oct 29 22:47:12.538441 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:12.538412 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-xf9r8"] Oct 29 22:47:12.549924 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:12.549900 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 29 22:47:12.549924 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:12.549910 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 29 22:47:12.630338 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:12.630306 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdlvg\" (UniqueName: \"kubernetes.io/projected/63bfe837-48d3-45b6-ae39-e4501036c02a-kube-api-access-rdlvg\") pod \"cert-manager-webhook-5dbbbf75cc-xf9r8\" (UID: \"63bfe837-48d3-45b6-ae39-e4501036c02a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xf9r8" Oct 29 22:47:12.630521 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:12.630368 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63bfe837-48d3-45b6-ae39-e4501036c02a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-xf9r8\" (UID: \"63bfe837-48d3-45b6-ae39-e4501036c02a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xf9r8" Oct 29 22:47:12.731337 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:12.731302 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdlvg\" (UniqueName: \"kubernetes.io/projected/63bfe837-48d3-45b6-ae39-e4501036c02a-kube-api-access-rdlvg\") pod \"cert-manager-webhook-5dbbbf75cc-xf9r8\" (UID: \"63bfe837-48d3-45b6-ae39-e4501036c02a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xf9r8" Oct 29 22:47:12.731496 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:12.731359 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63bfe837-48d3-45b6-ae39-e4501036c02a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-xf9r8\" (UID: \"63bfe837-48d3-45b6-ae39-e4501036c02a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xf9r8" Oct 29 22:47:12.770054 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:12.769989 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdlvg\" (UniqueName: \"kubernetes.io/projected/63bfe837-48d3-45b6-ae39-e4501036c02a-kube-api-access-rdlvg\") pod \"cert-manager-webhook-5dbbbf75cc-xf9r8\" (UID: \"63bfe837-48d3-45b6-ae39-e4501036c02a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xf9r8" Oct 29 22:47:12.770494 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:12.770467 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63bfe837-48d3-45b6-ae39-e4501036c02a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-xf9r8\" (UID: \"63bfe837-48d3-45b6-ae39-e4501036c02a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xf9r8" Oct 29 22:47:12.831818 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:12.831786 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xf9r8" Oct 29 22:47:13.137422 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:13.137394 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-xf9r8"] Oct 29 22:47:13.139064 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:47:13.139031 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63bfe837_48d3_45b6_ae39_e4501036c02a.slice/crio-a3bcbf47d3fc15fca1bfeff1c652cc031073df67c6d8033be1a9eef23d519072 WatchSource:0}: Error finding container a3bcbf47d3fc15fca1bfeff1c652cc031073df67c6d8033be1a9eef23d519072: Status 404 returned error can't find the container with id a3bcbf47d3fc15fca1bfeff1c652cc031073df67c6d8033be1a9eef23d519072 Oct 29 22:47:13.807985 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:13.807866 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xf9r8" event={"ID":"63bfe837-48d3-45b6-ae39-e4501036c02a","Type":"ContainerStarted","Data":"a3bcbf47d3fc15fca1bfeff1c652cc031073df67c6d8033be1a9eef23d519072"} Oct 29 22:47:16.827108 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:16.827071 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xf9r8" event={"ID":"63bfe837-48d3-45b6-ae39-e4501036c02a","Type":"ContainerStarted","Data":"cf4e37095d38099d839417385abd27ba51cf838470d1ba87eaab35a5664ed2fd"} Oct 29 22:47:16.827551 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:16.827140 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xf9r8" Oct 29 22:47:16.870052 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:16.869993 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xf9r8" podStartSLOduration=1.9352614209999999 podStartE2EDuration="4.869980913s" totalImagesPullingTime="2.934719492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:16.868881247 +0000 UTC m=+283.520379780" watchObservedRunningTime="2026-10-29 22:47:16.869980913 +0000 UTC m=+283.521479443" Oct 29 22:47:18.074304 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:18.074272 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:47:18.076709 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:18.076689 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-xhhgl" Oct 29 22:47:18.091735 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:18.091711 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 29 22:47:18.105524 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:18.105499 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:47:18.173258 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:18.173225 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:47:18.180176 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:18.180155 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctq2m\" (UniqueName: \"kubernetes.io/projected/7b0ec9a4-4f75-4119-9744-364ee7d69731-kube-api-access-ctq2m\") pod \"limitador-limitador-7c9b87d574-xhhgl\" (UID: \"7b0ec9a4-4f75-4119-9744-364ee7d69731\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xhhgl" Oct 29 22:47:18.180296 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:18.180277 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7b0ec9a4-4f75-4119-9744-364ee7d69731-config-file\") pod \"limitador-limitador-7c9b87d574-xhhgl\" (UID: \"7b0ec9a4-4f75-4119-9744-364ee7d69731\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xhhgl" Oct 29 22:47:18.281139 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:18.281105 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctq2m\" (UniqueName: \"kubernetes.io/projected/7b0ec9a4-4f75-4119-9744-364ee7d69731-kube-api-access-ctq2m\") pod \"limitador-limitador-7c9b87d574-xhhgl\" (UID: \"7b0ec9a4-4f75-4119-9744-364ee7d69731\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xhhgl" Oct 29 22:47:18.281312 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:18.281167 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7b0ec9a4-4f75-4119-9744-364ee7d69731-config-file\") pod \"limitador-limitador-7c9b87d574-xhhgl\" (UID: \"7b0ec9a4-4f75-4119-9744-364ee7d69731\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xhhgl" Oct 29 22:47:18.281748 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:18.281729 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7b0ec9a4-4f75-4119-9744-364ee7d69731-config-file\") pod \"limitador-limitador-7c9b87d574-xhhgl\" (UID: \"7b0ec9a4-4f75-4119-9744-364ee7d69731\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xhhgl" Oct 29 22:47:18.304846 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:18.304818 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctq2m\" (UniqueName: \"kubernetes.io/projected/7b0ec9a4-4f75-4119-9744-364ee7d69731-kube-api-access-ctq2m\") pod \"limitador-limitador-7c9b87d574-xhhgl\" (UID: \"7b0ec9a4-4f75-4119-9744-364ee7d69731\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xhhgl" Oct 29 22:47:18.388142 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:18.388062 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-xhhgl" Oct 29 22:47:18.682463 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:18.682431 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:47:18.683559 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:47:18.683530 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0ec9a4_4f75_4119_9744_364ee7d69731.slice/crio-d4f435a8f1f1415f5a9a002d290969cc8ef7c48dc67a30be857fc35ce827165b WatchSource:0}: Error finding container d4f435a8f1f1415f5a9a002d290969cc8ef7c48dc67a30be857fc35ce827165b: Status 404 returned error can't find the container with id d4f435a8f1f1415f5a9a002d290969cc8ef7c48dc67a30be857fc35ce827165b Oct 29 22:47:18.841866 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:18.841828 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-xhhgl" event={"ID":"7b0ec9a4-4f75-4119-9744-364ee7d69731","Type":"ContainerStarted","Data":"d4f435a8f1f1415f5a9a002d290969cc8ef7c48dc67a30be857fc35ce827165b"} Oct 29 22:47:18.896253 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:18.896223 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zw889"] Oct 29 22:47:18.899484 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:18.899467 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zw889" Oct 29 22:47:18.919142 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:18.919113 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xcpgt\"" Oct 29 22:47:18.921057 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:18.921029 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zw889"] Oct 29 22:47:18.987540 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:18.987462 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q6bh\" (UniqueName: \"kubernetes.io/projected/2afe0f3f-738f-409c-9e68-9b55f7e0e30a-kube-api-access-7q6bh\") pod \"authorino-56c6c9ddd5-zw889\" (UID: \"2afe0f3f-738f-409c-9e68-9b55f7e0e30a\") " pod="kuadrant-system/authorino-56c6c9ddd5-zw889" Oct 29 22:47:19.088521 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:19.088483 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q6bh\" (UniqueName: \"kubernetes.io/projected/2afe0f3f-738f-409c-9e68-9b55f7e0e30a-kube-api-access-7q6bh\") pod \"authorino-56c6c9ddd5-zw889\" (UID: \"2afe0f3f-738f-409c-9e68-9b55f7e0e30a\") " pod="kuadrant-system/authorino-56c6c9ddd5-zw889" Oct 29 22:47:19.112376 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:19.112340 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q6bh\" (UniqueName: \"kubernetes.io/projected/2afe0f3f-738f-409c-9e68-9b55f7e0e30a-kube-api-access-7q6bh\") pod \"authorino-56c6c9ddd5-zw889\" (UID: \"2afe0f3f-738f-409c-9e68-9b55f7e0e30a\") " pod="kuadrant-system/authorino-56c6c9ddd5-zw889" Oct 29 22:47:19.209746 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:19.209701 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zw889" Oct 29 22:47:19.508851 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:19.508821 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zw889"] Oct 29 22:47:19.509630 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:47:19.509601 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2afe0f3f_738f_409c_9e68_9b55f7e0e30a.slice/crio-bde1dbe99cd3eb0f6f5451c75cd2ce4cb852da202d0fcd8b0ba129945d267b8a WatchSource:0}: Error finding container bde1dbe99cd3eb0f6f5451c75cd2ce4cb852da202d0fcd8b0ba129945d267b8a: Status 404 returned error can't find the container with id bde1dbe99cd3eb0f6f5451c75cd2ce4cb852da202d0fcd8b0ba129945d267b8a Oct 29 22:47:19.847125 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:19.847032 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zw889" event={"ID":"2afe0f3f-738f-409c-9e68-9b55f7e0e30a","Type":"ContainerStarted","Data":"bde1dbe99cd3eb0f6f5451c75cd2ce4cb852da202d0fcd8b0ba129945d267b8a"} Oct 29 22:47:19.848581 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:19.848556 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-xhhgl" event={"ID":"7b0ec9a4-4f75-4119-9744-364ee7d69731","Type":"ContainerStarted","Data":"d1e4bf542acea19d23155a565f75d175d913a7521668330b32f8d87322069b47"} Oct 29 22:47:19.848726 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:19.848709 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-xhhgl" Oct 29 22:47:19.891238 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:19.891172 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-xhhgl" podStartSLOduration=1.038344392 podStartE2EDuration="1.891158286s" totalImagesPullingTime="852.813894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:19.889704845 +0000 UTC m=+286.541203385" watchObservedRunningTime="2026-10-29 22:47:19.891158286 +0000 UTC m=+286.542656815" Oct 29 22:47:21.859711 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:21.859667 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zw889" event={"ID":"2afe0f3f-738f-409c-9e68-9b55f7e0e30a","Type":"ContainerStarted","Data":"32de81d24ee5a1021f27eff4a81f274c91551c315b85ef469df3e119f392e8d0"} Oct 29 22:47:21.902705 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:21.902643 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-zw889" podStartSLOduration=2.473029827 podStartE2EDuration="3.90262717s" totalImagesPullingTime="1.429597343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:21.901005851 +0000 UTC m=+288.552504381" watchObservedRunningTime="2026-10-29 22:47:21.90262717 +0000 UTC m=+288.554125724" Oct 29 22:47:22.833733 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:22.833704 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xf9r8" Oct 29 22:47:30.854969 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:30.854938 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-xhhgl" Oct 29 22:47:33.499666 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:33.499637 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 22:47:33.500745 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:33.500722 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 22:47:33.911961 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:47:33.911876 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:47:55.314066 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.314024 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77cb9dc67b-67ptx"] Oct 29 22:47:55.319111 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.319088 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77cb9dc67b-67ptx" Oct 29 22:47:55.334680 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.334658 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 29 22:47:55.334796 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.334703 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 29 22:47:55.334851 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.334797 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 29 22:47:55.346937 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.346901 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 29 22:47:55.346937 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.346930 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6g56w\"" Oct 29 22:47:55.347142 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.347109 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 29 22:47:55.348985 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.348961 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77cb9dc67b-67ptx"] Oct 29 22:47:55.388770 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.388736 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e112bdf-3222-441d-8689-c8c133e6a93c-webhook-cert\") pod \"maas-controller-77cb9dc67b-67ptx\" (UID: \"9e112bdf-3222-441d-8689-c8c133e6a93c\") " pod="opendatahub/maas-controller-77cb9dc67b-67ptx" Oct 29 22:47:55.388955 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.388773 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e112bdf-3222-441d-8689-c8c133e6a93c-metrics-certs\") pod \"maas-controller-77cb9dc67b-67ptx\" (UID: \"9e112bdf-3222-441d-8689-c8c133e6a93c\") " pod="opendatahub/maas-controller-77cb9dc67b-67ptx" Oct 29 22:47:55.388955 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.388808 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dqgn\" (UniqueName: \"kubernetes.io/projected/9e112bdf-3222-441d-8689-c8c133e6a93c-kube-api-access-5dqgn\") pod \"maas-controller-77cb9dc67b-67ptx\" (UID: \"9e112bdf-3222-441d-8689-c8c133e6a93c\") " pod="opendatahub/maas-controller-77cb9dc67b-67ptx" Oct 29 22:47:55.489425 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.489397 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e112bdf-3222-441d-8689-c8c133e6a93c-webhook-cert\") pod \"maas-controller-77cb9dc67b-67ptx\" (UID: \"9e112bdf-3222-441d-8689-c8c133e6a93c\") " pod="opendatahub/maas-controller-77cb9dc67b-67ptx" Oct 29 22:47:55.489591 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.489433 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e112bdf-3222-441d-8689-c8c133e6a93c-metrics-certs\") pod \"maas-controller-77cb9dc67b-67ptx\" (UID: \"9e112bdf-3222-441d-8689-c8c133e6a93c\") " pod="opendatahub/maas-controller-77cb9dc67b-67ptx" Oct 29 22:47:55.489591 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.489477 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dqgn\" (UniqueName: \"kubernetes.io/projected/9e112bdf-3222-441d-8689-c8c133e6a93c-kube-api-access-5dqgn\") pod \"maas-controller-77cb9dc67b-67ptx\" (UID: \"9e112bdf-3222-441d-8689-c8c133e6a93c\") " pod="opendatahub/maas-controller-77cb9dc67b-67ptx" Oct 29 22:47:55.491921 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.491883 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e112bdf-3222-441d-8689-c8c133e6a93c-webhook-cert\") pod \"maas-controller-77cb9dc67b-67ptx\" (UID: \"9e112bdf-3222-441d-8689-c8c133e6a93c\") " pod="opendatahub/maas-controller-77cb9dc67b-67ptx" Oct 29 22:47:55.492048 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.491996 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e112bdf-3222-441d-8689-c8c133e6a93c-metrics-certs\") pod \"maas-controller-77cb9dc67b-67ptx\" (UID: \"9e112bdf-3222-441d-8689-c8c133e6a93c\") " pod="opendatahub/maas-controller-77cb9dc67b-67ptx" Oct 29 22:47:55.512996 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.512963 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dqgn\" (UniqueName: \"kubernetes.io/projected/9e112bdf-3222-441d-8689-c8c133e6a93c-kube-api-access-5dqgn\") pod \"maas-controller-77cb9dc67b-67ptx\" (UID: \"9e112bdf-3222-441d-8689-c8c133e6a93c\") " pod="opendatahub/maas-controller-77cb9dc67b-67ptx" Oct 29 22:47:55.630206 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.630116 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77cb9dc67b-67ptx" Oct 29 22:47:55.851711 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.851673 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6df9d4f548-fhxv7"] Oct 29 22:47:55.856216 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.856197 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6df9d4f548-fhxv7" Oct 29 22:47:55.878090 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.878063 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6df9d4f548-fhxv7"] Oct 29 22:47:55.892410 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.892350 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c748534c-ac5c-4bd0-98a0-51dc11412f9c-webhook-cert\") pod \"maas-controller-6df9d4f548-fhxv7\" (UID: \"c748534c-ac5c-4bd0-98a0-51dc11412f9c\") " pod="opendatahub/maas-controller-6df9d4f548-fhxv7" Oct 29 22:47:55.892410 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.892401 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz9mn\" (UniqueName: \"kubernetes.io/projected/c748534c-ac5c-4bd0-98a0-51dc11412f9c-kube-api-access-dz9mn\") pod \"maas-controller-6df9d4f548-fhxv7\" (UID: \"c748534c-ac5c-4bd0-98a0-51dc11412f9c\") " pod="opendatahub/maas-controller-6df9d4f548-fhxv7" Oct 29 22:47:55.892570 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.892481 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c748534c-ac5c-4bd0-98a0-51dc11412f9c-metrics-certs\") pod \"maas-controller-6df9d4f548-fhxv7\" (UID: \"c748534c-ac5c-4bd0-98a0-51dc11412f9c\") " pod="opendatahub/maas-controller-6df9d4f548-fhxv7" Oct 29 22:47:55.942848 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.942819 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77cb9dc67b-67ptx"] Oct 29 22:47:55.944464 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:47:55.944442 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e112bdf_3222_441d_8689_c8c133e6a93c.slice/crio-1d0f55df599bbc49af674bdf6426aad786e7e56d15dde54dd71beed4b9621e20 WatchSource:0}: Error finding container 1d0f55df599bbc49af674bdf6426aad786e7e56d15dde54dd71beed4b9621e20: Status 404 returned error can't find the container with id 1d0f55df599bbc49af674bdf6426aad786e7e56d15dde54dd71beed4b9621e20 Oct 29 22:47:55.945780 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.945763 5057 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:47:55.993852 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.993823 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c748534c-ac5c-4bd0-98a0-51dc11412f9c-webhook-cert\") pod \"maas-controller-6df9d4f548-fhxv7\" (UID: \"c748534c-ac5c-4bd0-98a0-51dc11412f9c\") " pod="opendatahub/maas-controller-6df9d4f548-fhxv7" Oct 29 22:47:55.993968 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.993874 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz9mn\" (UniqueName: \"kubernetes.io/projected/c748534c-ac5c-4bd0-98a0-51dc11412f9c-kube-api-access-dz9mn\") pod \"maas-controller-6df9d4f548-fhxv7\" (UID: \"c748534c-ac5c-4bd0-98a0-51dc11412f9c\") " pod="opendatahub/maas-controller-6df9d4f548-fhxv7" Oct 29 22:47:55.993968 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.993896 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c748534c-ac5c-4bd0-98a0-51dc11412f9c-metrics-certs\") pod \"maas-controller-6df9d4f548-fhxv7\" (UID: \"c748534c-ac5c-4bd0-98a0-51dc11412f9c\") " pod="opendatahub/maas-controller-6df9d4f548-fhxv7" Oct 29 22:47:55.996254 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.996227 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c748534c-ac5c-4bd0-98a0-51dc11412f9c-webhook-cert\") pod \"maas-controller-6df9d4f548-fhxv7\" (UID: \"c748534c-ac5c-4bd0-98a0-51dc11412f9c\") " pod="opendatahub/maas-controller-6df9d4f548-fhxv7" Oct 29 22:47:55.996254 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:55.996244 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c748534c-ac5c-4bd0-98a0-51dc11412f9c-metrics-certs\") pod \"maas-controller-6df9d4f548-fhxv7\" (UID: \"c748534c-ac5c-4bd0-98a0-51dc11412f9c\") " pod="opendatahub/maas-controller-6df9d4f548-fhxv7" Oct 29 22:47:56.017714 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:56.017687 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz9mn\" (UniqueName: \"kubernetes.io/projected/c748534c-ac5c-4bd0-98a0-51dc11412f9c-kube-api-access-dz9mn\") pod \"maas-controller-6df9d4f548-fhxv7\" (UID: \"c748534c-ac5c-4bd0-98a0-51dc11412f9c\") " pod="opendatahub/maas-controller-6df9d4f548-fhxv7" Oct 29 22:47:56.018692 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:56.018667 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77cb9dc67b-67ptx" event={"ID":"9e112bdf-3222-441d-8689-c8c133e6a93c","Type":"ContainerStarted","Data":"1d0f55df599bbc49af674bdf6426aad786e7e56d15dde54dd71beed4b9621e20"} Oct 29 22:47:56.169650 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:56.169573 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6df9d4f548-fhxv7" Oct 29 22:47:56.473323 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:56.473278 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6df9d4f548-fhxv7"] Oct 29 22:47:56.477581 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:47:56.477556 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc748534c_ac5c_4bd0_98a0_51dc11412f9c.slice/crio-187c5ca2af24c322387616e97b8ef42f1ceef0dc0787c769d68cece8261b55f5 WatchSource:0}: Error finding container 187c5ca2af24c322387616e97b8ef42f1ceef0dc0787c769d68cece8261b55f5: Status 404 returned error can't find the container with id 187c5ca2af24c322387616e97b8ef42f1ceef0dc0787c769d68cece8261b55f5 Oct 29 22:47:57.027461 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:57.027415 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6df9d4f548-fhxv7" event={"ID":"c748534c-ac5c-4bd0-98a0-51dc11412f9c","Type":"ContainerStarted","Data":"187c5ca2af24c322387616e97b8ef42f1ceef0dc0787c769d68cece8261b55f5"} Oct 29 22:47:59.038646 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:59.038607 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6df9d4f548-fhxv7" event={"ID":"c748534c-ac5c-4bd0-98a0-51dc11412f9c","Type":"ContainerStarted","Data":"aaebe998447193dff70facff3c440c12ae023952ca6bb64c5c32fcda35302f66"} Oct 29 22:47:59.039095 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:59.038673 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6df9d4f548-fhxv7" Oct 29 22:47:59.040165 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:59.040139 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77cb9dc67b-67ptx" event={"ID":"9e112bdf-3222-441d-8689-c8c133e6a93c","Type":"ContainerStarted","Data":"eef8cfcbbc0df4e20234ecd230bfd89e0fd792f75bec1e858c09c154c1b76b3f"} Oct 29 22:47:59.040301 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:59.040220 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77cb9dc67b-67ptx" Oct 29 22:47:59.092131 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:47:59.092073 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6df9d4f548-fhxv7" podStartSLOduration=2.018852513 podStartE2EDuration="4.092057099s" totalImagesPullingTime="2.073204586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:59.089517643 +0000 UTC m=+325.741016176" watchObservedRunningTime="2026-10-29 22:47:59.092057099 +0000 UTC m=+325.743555629" Oct 29 22:48:03.913280 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:48:03.913242 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:48:05.496545 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.496486 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-77cb9dc67b-67ptx" podStartSLOduration=7.899262316 podStartE2EDuration="10.496472849s" totalImagesPullingTime="2.597210533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:59.145176587 +0000 UTC m=+325.796675128" watchObservedRunningTime="2026-10-29 22:48:05.496472849 +0000 UTC m=+332.147971380" Oct 29 22:48:05.496957 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.496629 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw"] Oct 29 22:48:05.500287 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.500266 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.517139 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.517115 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 29 22:48:05.517369 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.517353 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 29 22:48:05.517456 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.517368 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 29 22:48:05.517456 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.517442 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7dsjj\"" Oct 29 22:48:05.535608 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.535577 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw"] Oct 29 22:48:05.587613 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.587580 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.587613 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.587615 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znwxp\" (UniqueName: \"kubernetes.io/projected/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-kube-api-access-znwxp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.587814 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.587644 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.587814 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.587775 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.587887 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.587817 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.587887 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.587835 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.587964 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.587920 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.689204 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.689160 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.689373 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.689242 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.689373 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.689289 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.689373 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.689316 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.689539 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.689377 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.689539 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.689412 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.689539 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.689441 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znwxp\" (UniqueName: \"kubernetes.io/projected/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-kube-api-access-znwxp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.689942 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.689914 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.691692 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.691658 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.691802 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.691769 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.691900 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.691879 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.692011 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.691995 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.714613 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.714591 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.714801 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.714781 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znwxp\" (UniqueName: \"kubernetes.io/projected/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-kube-api-access-znwxp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vf9zw\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:05.812073 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:05.811999 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:06.149810 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:06.149781 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw"] Oct 29 22:48:06.151668 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:48:06.151633 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b5b01f_807f_4fa6_b4aa_b1c338e2e3df.slice/crio-68262d7ed1ba6ec1da67fe324c443feaba5853146ad787c910c82dc889aabe75 WatchSource:0}: Error finding container 68262d7ed1ba6ec1da67fe324c443feaba5853146ad787c910c82dc889aabe75: Status 404 returned error can't find the container with id 68262d7ed1ba6ec1da67fe324c443feaba5853146ad787c910c82dc889aabe75 Oct 29 22:48:07.079767 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:07.079731 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" event={"ID":"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df","Type":"ContainerStarted","Data":"68262d7ed1ba6ec1da67fe324c443feaba5853146ad787c910c82dc889aabe75"} Oct 29 22:48:08.539712 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:08.539681 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:48:08.539950 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:08.539745 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:48:09.092092 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:09.092054 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" event={"ID":"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df","Type":"ContainerStarted","Data":"7771cc1187adf7c5ce7b83c538425f60a21ad8a614f9c614a7c39d4bcaf9f93b"} Oct 29 22:48:09.092342 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:09.092323 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:09.094036 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:09.094011 5057 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vf9zw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:48:09.094132 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:09.094056 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" podUID="88b5b01f-807f-4fa6-b4aa-b1c338e2e3df" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:48:09.172014 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:09.171960 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" podStartSLOduration=1.786879873 podStartE2EDuration="4.171947404s" totalImagesPullingTime="2.385067531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:09.15664548 +0000 UTC m=+335.808144011" watchObservedRunningTime="2026-10-29 22:48:09.171947404 +0000 UTC m=+335.823445926" Oct 29 22:48:10.047451 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:10.047421 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6df9d4f548-fhxv7" Oct 29 22:48:10.047825 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:10.047490 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77cb9dc67b-67ptx" Oct 29 22:48:10.097753 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:10.097725 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 22:48:10.125336 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:10.125302 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77cb9dc67b-67ptx"] Oct 29 22:48:10.125533 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:10.125509 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-77cb9dc67b-67ptx" podUID="9e112bdf-3222-441d-8689-c8c133e6a93c" containerName="manager" containerID="cri-o://eef8cfcbbc0df4e20234ecd230bfd89e0fd792f75bec1e858c09c154c1b76b3f" gracePeriod=30 Oct 29 22:48:10.377973 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:10.377950 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77cb9dc67b-67ptx" Oct 29 22:48:10.437139 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:10.437112 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e112bdf-3222-441d-8689-c8c133e6a93c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e112bdf-3222-441d-8689-c8c133e6a93c-metrics-certs\") pod \"9e112bdf-3222-441d-8689-c8c133e6a93c\" (UID: \"9e112bdf-3222-441d-8689-c8c133e6a93c\") " Oct 29 22:48:10.437319 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:10.437157 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e112bdf-3222-441d-8689-c8c133e6a93c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e112bdf-3222-441d-8689-c8c133e6a93c-webhook-cert\") pod \"9e112bdf-3222-441d-8689-c8c133e6a93c\" (UID: \"9e112bdf-3222-441d-8689-c8c133e6a93c\") " Oct 29 22:48:10.437319 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:10.437277 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e112bdf-3222-441d-8689-c8c133e6a93c-kube-api-access-5dqgn\" (UniqueName: \"kubernetes.io/projected/9e112bdf-3222-441d-8689-c8c133e6a93c-kube-api-access-5dqgn\") pod \"9e112bdf-3222-441d-8689-c8c133e6a93c\" (UID: \"9e112bdf-3222-441d-8689-c8c133e6a93c\") " Oct 29 22:48:10.439460 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:10.439427 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e112bdf-3222-441d-8689-c8c133e6a93c-webhook-cert" pod "9e112bdf-3222-441d-8689-c8c133e6a93c" (UID: "9e112bdf-3222-441d-8689-c8c133e6a93c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:10.439565 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:10.439491 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e112bdf-3222-441d-8689-c8c133e6a93c-kube-api-access-5dqgn" pod "9e112bdf-3222-441d-8689-c8c133e6a93c" (UID: "9e112bdf-3222-441d-8689-c8c133e6a93c"). InnerVolumeSpecName "kube-api-access-5dqgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:10.439565 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:10.439491 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e112bdf-3222-441d-8689-c8c133e6a93c-metrics-certs" pod "9e112bdf-3222-441d-8689-c8c133e6a93c" (UID: "9e112bdf-3222-441d-8689-c8c133e6a93c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:10.538134 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:10.538091 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dqgn\" (UniqueName: \"kubernetes.io/projected/9e112bdf-3222-441d-8689-c8c133e6a93c-kube-api-access-5dqgn\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:48:10.538134 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:10.538138 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e112bdf-3222-441d-8689-c8c133e6a93c-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:48:10.538352 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:10.538153 5057 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e112bdf-3222-441d-8689-c8c133e6a93c-webhook-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:48:11.103357 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:11.103326 5057 generic.go:356] "Generic (PLEG): container finished" podID="9e112bdf-3222-441d-8689-c8c133e6a93c" containerID="eef8cfcbbc0df4e20234ecd230bfd89e0fd792f75bec1e858c09c154c1b76b3f" exitCode=0 Oct 29 22:48:11.103800 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:11.103374 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77cb9dc67b-67ptx" Oct 29 22:48:11.103800 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:11.103416 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77cb9dc67b-67ptx" event={"ID":"9e112bdf-3222-441d-8689-c8c133e6a93c","Type":"ContainerDied","Data":"eef8cfcbbc0df4e20234ecd230bfd89e0fd792f75bec1e858c09c154c1b76b3f"} Oct 29 22:48:11.103800 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:11.103468 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77cb9dc67b-67ptx" event={"ID":"9e112bdf-3222-441d-8689-c8c133e6a93c","Type":"ContainerDied","Data":"1d0f55df599bbc49af674bdf6426aad786e7e56d15dde54dd71beed4b9621e20"} Oct 29 22:48:11.103800 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:11.103491 5057 scope.go:122] "RemoveContainer" containerID="eef8cfcbbc0df4e20234ecd230bfd89e0fd792f75bec1e858c09c154c1b76b3f" Oct 29 22:48:11.118732 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:11.118711 5057 scope.go:122] "RemoveContainer" containerID="eef8cfcbbc0df4e20234ecd230bfd89e0fd792f75bec1e858c09c154c1b76b3f" Oct 29 22:48:11.119037 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:48:11.119018 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eef8cfcbbc0df4e20234ecd230bfd89e0fd792f75bec1e858c09c154c1b76b3f\": container with ID starting with eef8cfcbbc0df4e20234ecd230bfd89e0fd792f75bec1e858c09c154c1b76b3f not found: ID does not exist" containerID="eef8cfcbbc0df4e20234ecd230bfd89e0fd792f75bec1e858c09c154c1b76b3f" Oct 29 22:48:11.119083 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:11.119047 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef8cfcbbc0df4e20234ecd230bfd89e0fd792f75bec1e858c09c154c1b76b3f"} err="failed to get container status \"eef8cfcbbc0df4e20234ecd230bfd89e0fd792f75bec1e858c09c154c1b76b3f\": rpc error: code = NotFound desc = could not find container \"eef8cfcbbc0df4e20234ecd230bfd89e0fd792f75bec1e858c09c154c1b76b3f\": container with ID starting with eef8cfcbbc0df4e20234ecd230bfd89e0fd792f75bec1e858c09c154c1b76b3f not found: ID does not exist" Oct 29 22:48:11.178265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:11.178236 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77cb9dc67b-67ptx"] Oct 29 22:48:11.192462 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:11.192434 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77cb9dc67b-67ptx"] Oct 29 22:48:11.482011 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:11.481980 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e112bdf-3222-441d-8689-c8c133e6a93c" path="/var/lib/kubelet/pods/9e112bdf-3222-441d-8689-c8c133e6a93c/volumes" Oct 29 22:48:32.849772 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:32.849726 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:48:33.912942 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:48:33.912863 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:48:44.501410 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:44.501376 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zw889"] Oct 29 22:48:44.501893 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:44.501556 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-zw889" podUID="2afe0f3f-738f-409c-9e68-9b55f7e0e30a" containerName="authorino" containerID="cri-o://32de81d24ee5a1021f27eff4a81f274c91551c315b85ef469df3e119f392e8d0" gracePeriod=30 Oct 29 22:48:44.755764 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:44.755703 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zw889" Oct 29 22:48:44.950138 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:44.950102 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2afe0f3f-738f-409c-9e68-9b55f7e0e30a-kube-api-access-7q6bh\" (UniqueName: \"kubernetes.io/projected/2afe0f3f-738f-409c-9e68-9b55f7e0e30a-kube-api-access-7q6bh\") pod \"2afe0f3f-738f-409c-9e68-9b55f7e0e30a\" (UID: \"2afe0f3f-738f-409c-9e68-9b55f7e0e30a\") " Oct 29 22:48:44.952387 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:44.952363 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2afe0f3f-738f-409c-9e68-9b55f7e0e30a-kube-api-access-7q6bh" pod "2afe0f3f-738f-409c-9e68-9b55f7e0e30a" (UID: "2afe0f3f-738f-409c-9e68-9b55f7e0e30a"). InnerVolumeSpecName "kube-api-access-7q6bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:45.051587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:45.051496 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q6bh\" (UniqueName: \"kubernetes.io/projected/2afe0f3f-738f-409c-9e68-9b55f7e0e30a-kube-api-access-7q6bh\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:48:45.259739 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:45.259701 5057 generic.go:356] "Generic (PLEG): container finished" podID="2afe0f3f-738f-409c-9e68-9b55f7e0e30a" containerID="32de81d24ee5a1021f27eff4a81f274c91551c315b85ef469df3e119f392e8d0" exitCode=0 Oct 29 22:48:45.259918 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:45.259750 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zw889" Oct 29 22:48:45.259918 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:45.259775 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zw889" event={"ID":"2afe0f3f-738f-409c-9e68-9b55f7e0e30a","Type":"ContainerDied","Data":"32de81d24ee5a1021f27eff4a81f274c91551c315b85ef469df3e119f392e8d0"} Oct 29 22:48:45.259918 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:45.259824 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zw889" event={"ID":"2afe0f3f-738f-409c-9e68-9b55f7e0e30a","Type":"ContainerDied","Data":"bde1dbe99cd3eb0f6f5451c75cd2ce4cb852da202d0fcd8b0ba129945d267b8a"} Oct 29 22:48:45.259918 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:45.259841 5057 scope.go:122] "RemoveContainer" containerID="32de81d24ee5a1021f27eff4a81f274c91551c315b85ef469df3e119f392e8d0" Oct 29 22:48:45.268313 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:45.268294 5057 scope.go:122] "RemoveContainer" containerID="32de81d24ee5a1021f27eff4a81f274c91551c315b85ef469df3e119f392e8d0" Oct 29 22:48:45.268555 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:48:45.268536 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32de81d24ee5a1021f27eff4a81f274c91551c315b85ef469df3e119f392e8d0\": container with ID starting with 32de81d24ee5a1021f27eff4a81f274c91551c315b85ef469df3e119f392e8d0 not found: ID does not exist" containerID="32de81d24ee5a1021f27eff4a81f274c91551c315b85ef469df3e119f392e8d0" Oct 29 22:48:45.268616 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:45.268564 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32de81d24ee5a1021f27eff4a81f274c91551c315b85ef469df3e119f392e8d0"} err="failed to get container status \"32de81d24ee5a1021f27eff4a81f274c91551c315b85ef469df3e119f392e8d0\": rpc error: code = NotFound desc = could not find container \"32de81d24ee5a1021f27eff4a81f274c91551c315b85ef469df3e119f392e8d0\": container with ID starting with 32de81d24ee5a1021f27eff4a81f274c91551c315b85ef469df3e119f392e8d0 not found: ID does not exist" Oct 29 22:48:45.316350 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:45.316266 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zw889"] Oct 29 22:48:45.330441 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:45.330412 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zw889"] Oct 29 22:48:45.484933 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:48:45.484899 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2afe0f3f-738f-409c-9e68-9b55f7e0e30a" path="/var/lib/kubelet/pods/2afe0f3f-738f-409c-9e68-9b55f7e0e30a/volumes" Oct 29 22:49:03.911980 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:49:03.911944 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:49:33.912506 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:49:33.912417 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:50:03.912387 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:50:03.912352 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:50:28.734167 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:28.734134 5057 generic.go:356] "Generic (PLEG): container finished" podID="c748534c-ac5c-4bd0-98a0-51dc11412f9c" containerID="aaebe998447193dff70facff3c440c12ae023952ca6bb64c5c32fcda35302f66" exitCode=1 Oct 29 22:50:28.734602 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:28.734210 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6df9d4f548-fhxv7" event={"ID":"c748534c-ac5c-4bd0-98a0-51dc11412f9c","Type":"ContainerDied","Data":"aaebe998447193dff70facff3c440c12ae023952ca6bb64c5c32fcda35302f66"} Oct 29 22:50:28.734602 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:28.734565 5057 scope.go:122] "RemoveContainer" containerID="aaebe998447193dff70facff3c440c12ae023952ca6bb64c5c32fcda35302f66" Oct 29 22:50:29.740738 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:29.740703 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6df9d4f548-fhxv7" event={"ID":"c748534c-ac5c-4bd0-98a0-51dc11412f9c","Type":"ContainerStarted","Data":"89ae88f5f3076ef46e8deb1c2bdc768faf9f4485199cffca3c557cfa1f671f70"} Oct 29 22:50:29.741120 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:29.740921 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6df9d4f548-fhxv7" Oct 29 22:50:32.952315 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:32.952278 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:50:33.911623 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:50:33.911588 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:50:35.756987 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.756958 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv"] Oct 29 22:50:35.757493 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.757336 5057 cpu_manager.go:414] "containerMap: removing container" podUID="2afe0f3f-738f-409c-9e68-9b55f7e0e30a" containerName="authorino" Oct 29 22:50:35.757493 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.757349 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2afe0f3f-738f-409c-9e68-9b55f7e0e30a" containerName="authorino" Oct 29 22:50:35.757493 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.757377 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9e112bdf-3222-441d-8689-c8c133e6a93c" containerName="manager" Oct 29 22:50:35.757493 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.757382 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e112bdf-3222-441d-8689-c8c133e6a93c" containerName="manager" Oct 29 22:50:35.757493 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.757463 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e112bdf-3222-441d-8689-c8c133e6a93c" containerName="manager" Oct 29 22:50:35.757493 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.757474 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="2afe0f3f-738f-409c-9e68-9b55f7e0e30a" containerName="authorino" Oct 29 22:50:35.760020 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.760000 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:35.777064 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.777038 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 29 22:50:35.790926 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.790895 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 29 22:50:35.791208 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.791165 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 29 22:50:35.791391 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.791368 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-28mq9\"" Oct 29 22:50:35.793035 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.793008 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv"] Oct 29 22:50:35.820451 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.820427 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/907c36dd-72ba-42b8-86f1-9ff84400b4b2-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2qkxv\" (UID: \"907c36dd-72ba-42b8-86f1-9ff84400b4b2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:35.820560 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.820492 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/907c36dd-72ba-42b8-86f1-9ff84400b4b2-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2qkxv\" (UID: \"907c36dd-72ba-42b8-86f1-9ff84400b4b2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:35.820601 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.820572 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/907c36dd-72ba-42b8-86f1-9ff84400b4b2-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2qkxv\" (UID: \"907c36dd-72ba-42b8-86f1-9ff84400b4b2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:35.820637 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.820601 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/907c36dd-72ba-42b8-86f1-9ff84400b4b2-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2qkxv\" (UID: \"907c36dd-72ba-42b8-86f1-9ff84400b4b2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:35.820637 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.820623 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/907c36dd-72ba-42b8-86f1-9ff84400b4b2-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2qkxv\" (UID: \"907c36dd-72ba-42b8-86f1-9ff84400b4b2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:35.820698 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.820668 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rvv7\" (UniqueName: \"kubernetes.io/projected/907c36dd-72ba-42b8-86f1-9ff84400b4b2-kube-api-access-5rvv7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2qkxv\" (UID: \"907c36dd-72ba-42b8-86f1-9ff84400b4b2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:35.921375 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.921342 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/907c36dd-72ba-42b8-86f1-9ff84400b4b2-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2qkxv\" (UID: \"907c36dd-72ba-42b8-86f1-9ff84400b4b2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:35.921563 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.921388 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/907c36dd-72ba-42b8-86f1-9ff84400b4b2-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2qkxv\" (UID: \"907c36dd-72ba-42b8-86f1-9ff84400b4b2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:35.921563 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.921409 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/907c36dd-72ba-42b8-86f1-9ff84400b4b2-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2qkxv\" (UID: \"907c36dd-72ba-42b8-86f1-9ff84400b4b2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:35.921563 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.921515 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/907c36dd-72ba-42b8-86f1-9ff84400b4b2-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2qkxv\" (UID: \"907c36dd-72ba-42b8-86f1-9ff84400b4b2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:35.921733 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.921559 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rvv7\" (UniqueName: \"kubernetes.io/projected/907c36dd-72ba-42b8-86f1-9ff84400b4b2-kube-api-access-5rvv7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2qkxv\" (UID: \"907c36dd-72ba-42b8-86f1-9ff84400b4b2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:35.921733 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.921619 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/907c36dd-72ba-42b8-86f1-9ff84400b4b2-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2qkxv\" (UID: \"907c36dd-72ba-42b8-86f1-9ff84400b4b2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:35.921845 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.921750 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/907c36dd-72ba-42b8-86f1-9ff84400b4b2-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2qkxv\" (UID: \"907c36dd-72ba-42b8-86f1-9ff84400b4b2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:35.921845 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.921822 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/907c36dd-72ba-42b8-86f1-9ff84400b4b2-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2qkxv\" (UID: \"907c36dd-72ba-42b8-86f1-9ff84400b4b2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:35.921936 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.921886 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/907c36dd-72ba-42b8-86f1-9ff84400b4b2-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2qkxv\" (UID: \"907c36dd-72ba-42b8-86f1-9ff84400b4b2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:35.923901 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.923878 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/907c36dd-72ba-42b8-86f1-9ff84400b4b2-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2qkxv\" (UID: \"907c36dd-72ba-42b8-86f1-9ff84400b4b2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:35.924054 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.924040 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/907c36dd-72ba-42b8-86f1-9ff84400b4b2-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2qkxv\" (UID: \"907c36dd-72ba-42b8-86f1-9ff84400b4b2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:35.947642 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:35.947559 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rvv7\" (UniqueName: \"kubernetes.io/projected/907c36dd-72ba-42b8-86f1-9ff84400b4b2-kube-api-access-5rvv7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2qkxv\" (UID: \"907c36dd-72ba-42b8-86f1-9ff84400b4b2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:36.076416 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:36.076326 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:36.371620 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:36.371593 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv"] Oct 29 22:50:36.373092 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:50:36.373068 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907c36dd_72ba_42b8_86f1_9ff84400b4b2.slice/crio-35a266ffaeffd9c75e12c7c3d58271d355c93257a6101a9edfb935613eed9997 WatchSource:0}: Error finding container 35a266ffaeffd9c75e12c7c3d58271d355c93257a6101a9edfb935613eed9997: Status 404 returned error can't find the container with id 35a266ffaeffd9c75e12c7c3d58271d355c93257a6101a9edfb935613eed9997 Oct 29 22:50:36.776393 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:36.776356 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" event={"ID":"907c36dd-72ba-42b8-86f1-9ff84400b4b2","Type":"ContainerStarted","Data":"35a266ffaeffd9c75e12c7c3d58271d355c93257a6101a9edfb935613eed9997"} Oct 29 22:50:36.996027 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:36.995986 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:50:37.783283 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:37.783205 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" event={"ID":"907c36dd-72ba-42b8-86f1-9ff84400b4b2","Type":"ContainerStarted","Data":"d47d5849e3b3c0324cfad89d11a291ae4ffc7a81eac4a92f073f07e9b8cc3665"} Oct 29 22:50:37.830245 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:37.830176 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" podStartSLOduration=1.740607421 podStartE2EDuration="2.830164177s" totalImagesPullingTime="1.089556756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:37.829433035 +0000 UTC m=+484.480931565" watchObservedRunningTime="2026-10-29 22:50:37.830164177 +0000 UTC m=+484.481662707" Oct 29 22:50:38.077068 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.076979 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:38.766306 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.766274 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l"] Oct 29 22:50:38.770285 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.770257 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:38.790405 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.790382 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 29 22:50:38.805974 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.805937 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l"] Oct 29 22:50:38.849608 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.849572 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce69276-30ac-47e6-99df-81608424b3dd-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7ks2l\" (UID: \"bce69276-30ac-47e6-99df-81608424b3dd\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:38.849779 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.849624 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drlzs\" (UniqueName: \"kubernetes.io/projected/bce69276-30ac-47e6-99df-81608424b3dd-kube-api-access-drlzs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7ks2l\" (UID: \"bce69276-30ac-47e6-99df-81608424b3dd\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:38.849779 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.849725 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce69276-30ac-47e6-99df-81608424b3dd-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7ks2l\" (UID: \"bce69276-30ac-47e6-99df-81608424b3dd\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:38.849779 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.849751 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce69276-30ac-47e6-99df-81608424b3dd-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7ks2l\" (UID: \"bce69276-30ac-47e6-99df-81608424b3dd\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:38.849907 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.849865 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce69276-30ac-47e6-99df-81608424b3dd-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7ks2l\" (UID: \"bce69276-30ac-47e6-99df-81608424b3dd\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:38.849998 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.849983 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce69276-30ac-47e6-99df-81608424b3dd-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7ks2l\" (UID: \"bce69276-30ac-47e6-99df-81608424b3dd\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:38.950606 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.950569 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce69276-30ac-47e6-99df-81608424b3dd-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7ks2l\" (UID: \"bce69276-30ac-47e6-99df-81608424b3dd\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:38.950606 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.950604 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce69276-30ac-47e6-99df-81608424b3dd-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7ks2l\" (UID: \"bce69276-30ac-47e6-99df-81608424b3dd\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:38.950851 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.950630 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce69276-30ac-47e6-99df-81608424b3dd-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7ks2l\" (UID: \"bce69276-30ac-47e6-99df-81608424b3dd\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:38.950851 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.950769 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce69276-30ac-47e6-99df-81608424b3dd-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7ks2l\" (UID: \"bce69276-30ac-47e6-99df-81608424b3dd\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:38.950851 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.950811 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce69276-30ac-47e6-99df-81608424b3dd-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7ks2l\" (UID: \"bce69276-30ac-47e6-99df-81608424b3dd\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:38.950851 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.950837 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drlzs\" (UniqueName: \"kubernetes.io/projected/bce69276-30ac-47e6-99df-81608424b3dd-kube-api-access-drlzs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7ks2l\" (UID: \"bce69276-30ac-47e6-99df-81608424b3dd\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:38.951091 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.951061 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce69276-30ac-47e6-99df-81608424b3dd-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7ks2l\" (UID: \"bce69276-30ac-47e6-99df-81608424b3dd\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:38.951144 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.951096 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce69276-30ac-47e6-99df-81608424b3dd-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7ks2l\" (UID: \"bce69276-30ac-47e6-99df-81608424b3dd\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:38.951272 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.951230 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce69276-30ac-47e6-99df-81608424b3dd-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7ks2l\" (UID: \"bce69276-30ac-47e6-99df-81608424b3dd\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:38.953531 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.953506 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce69276-30ac-47e6-99df-81608424b3dd-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7ks2l\" (UID: \"bce69276-30ac-47e6-99df-81608424b3dd\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:38.953875 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.953853 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce69276-30ac-47e6-99df-81608424b3dd-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7ks2l\" (UID: \"bce69276-30ac-47e6-99df-81608424b3dd\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:38.977539 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:38.977511 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drlzs\" (UniqueName: \"kubernetes.io/projected/bce69276-30ac-47e6-99df-81608424b3dd-kube-api-access-drlzs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7ks2l\" (UID: \"bce69276-30ac-47e6-99df-81608424b3dd\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:39.083428 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:39.083351 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:39.388126 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:39.388097 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l"] Oct 29 22:50:39.389796 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:50:39.389770 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce69276_30ac_47e6_99df_81608424b3dd.slice/crio-773b48b51e8c76d0282d05c1585bf6f24022ee28bfcf977e29b5fcff3d75b061 WatchSource:0}: Error finding container 773b48b51e8c76d0282d05c1585bf6f24022ee28bfcf977e29b5fcff3d75b061: Status 404 returned error can't find the container with id 773b48b51e8c76d0282d05c1585bf6f24022ee28bfcf977e29b5fcff3d75b061 Oct 29 22:50:39.795089 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:39.795050 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" event={"ID":"bce69276-30ac-47e6-99df-81608424b3dd","Type":"ContainerStarted","Data":"02b9815713ebb2a2a03ec13f7e731fcb95eb0d8fac4d40c0005d2cefb93029d4"} Oct 29 22:50:39.795486 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:39.795104 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" event={"ID":"bce69276-30ac-47e6-99df-81608424b3dd","Type":"ContainerStarted","Data":"773b48b51e8c76d0282d05c1585bf6f24022ee28bfcf977e29b5fcff3d75b061"} Oct 29 22:50:39.845882 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:39.845828 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" podStartSLOduration=1.635608898 podStartE2EDuration="1.845813828s" totalImagesPullingTime="210.20493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:39.844156866 +0000 UTC m=+486.495655397" watchObservedRunningTime="2026-10-29 22:50:39.845813828 +0000 UTC m=+486.497312358" Oct 29 22:50:40.083703 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:40.083667 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:40.748531 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:40.748496 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6df9d4f548-fhxv7" Oct 29 22:50:40.846815 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:40.846783 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:50:42.806808 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.806772 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-5m4cw"] Oct 29 22:50:42.811392 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.811366 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:42.828511 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.828486 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 29 22:50:42.843981 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.843952 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-5m4cw"] Oct 29 22:50:42.884673 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.884642 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7503b3-a2b7-4795-b78a-9f18d00b5f2f-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-5m4cw\" (UID: \"5f7503b3-a2b7-4795-b78a-9f18d00b5f2f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:42.884879 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.884786 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f7503b3-a2b7-4795-b78a-9f18d00b5f2f-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-5m4cw\" (UID: \"5f7503b3-a2b7-4795-b78a-9f18d00b5f2f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:42.884879 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.884822 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f7503b3-a2b7-4795-b78a-9f18d00b5f2f-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-5m4cw\" (UID: \"5f7503b3-a2b7-4795-b78a-9f18d00b5f2f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:42.884879 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.884856 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cxqf\" (UniqueName: \"kubernetes.io/projected/5f7503b3-a2b7-4795-b78a-9f18d00b5f2f-kube-api-access-2cxqf\") pod \"echo-simulator-kserve-64b6f8f96c-5m4cw\" (UID: \"5f7503b3-a2b7-4795-b78a-9f18d00b5f2f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:42.885024 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.884884 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f7503b3-a2b7-4795-b78a-9f18d00b5f2f-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-5m4cw\" (UID: \"5f7503b3-a2b7-4795-b78a-9f18d00b5f2f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:42.885083 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.885064 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f7503b3-a2b7-4795-b78a-9f18d00b5f2f-home\") pod \"echo-simulator-kserve-64b6f8f96c-5m4cw\" (UID: \"5f7503b3-a2b7-4795-b78a-9f18d00b5f2f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:42.986320 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.986269 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f7503b3-a2b7-4795-b78a-9f18d00b5f2f-home\") pod \"echo-simulator-kserve-64b6f8f96c-5m4cw\" (UID: \"5f7503b3-a2b7-4795-b78a-9f18d00b5f2f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:42.986496 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.986336 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7503b3-a2b7-4795-b78a-9f18d00b5f2f-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-5m4cw\" (UID: \"5f7503b3-a2b7-4795-b78a-9f18d00b5f2f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:42.986496 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.986412 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f7503b3-a2b7-4795-b78a-9f18d00b5f2f-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-5m4cw\" (UID: \"5f7503b3-a2b7-4795-b78a-9f18d00b5f2f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:42.986496 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.986453 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f7503b3-a2b7-4795-b78a-9f18d00b5f2f-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-5m4cw\" (UID: \"5f7503b3-a2b7-4795-b78a-9f18d00b5f2f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:42.986496 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.986479 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cxqf\" (UniqueName: \"kubernetes.io/projected/5f7503b3-a2b7-4795-b78a-9f18d00b5f2f-kube-api-access-2cxqf\") pod \"echo-simulator-kserve-64b6f8f96c-5m4cw\" (UID: \"5f7503b3-a2b7-4795-b78a-9f18d00b5f2f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:42.986699 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.986505 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f7503b3-a2b7-4795-b78a-9f18d00b5f2f-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-5m4cw\" (UID: \"5f7503b3-a2b7-4795-b78a-9f18d00b5f2f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:42.986813 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.986789 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f7503b3-a2b7-4795-b78a-9f18d00b5f2f-home\") pod \"echo-simulator-kserve-64b6f8f96c-5m4cw\" (UID: \"5f7503b3-a2b7-4795-b78a-9f18d00b5f2f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:42.986928 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.986905 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f7503b3-a2b7-4795-b78a-9f18d00b5f2f-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-5m4cw\" (UID: \"5f7503b3-a2b7-4795-b78a-9f18d00b5f2f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:42.986995 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.986966 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f7503b3-a2b7-4795-b78a-9f18d00b5f2f-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-5m4cw\" (UID: \"5f7503b3-a2b7-4795-b78a-9f18d00b5f2f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:42.988730 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.988701 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f7503b3-a2b7-4795-b78a-9f18d00b5f2f-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-5m4cw\" (UID: \"5f7503b3-a2b7-4795-b78a-9f18d00b5f2f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:42.988976 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:42.988959 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7503b3-a2b7-4795-b78a-9f18d00b5f2f-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-5m4cw\" (UID: \"5f7503b3-a2b7-4795-b78a-9f18d00b5f2f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:43.015776 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:43.015749 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cxqf\" (UniqueName: \"kubernetes.io/projected/5f7503b3-a2b7-4795-b78a-9f18d00b5f2f-kube-api-access-2cxqf\") pod \"echo-simulator-kserve-64b6f8f96c-5m4cw\" (UID: \"5f7503b3-a2b7-4795-b78a-9f18d00b5f2f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:43.126853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:43.126764 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:43.433990 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:43.433966 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-5m4cw"] Oct 29 22:50:43.435359 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:50:43.435331 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7503b3_a2b7_4795_b78a_9f18d00b5f2f.slice/crio-05f79ec45d5461b74490185c1e53588c7cf3736a8320fb7b8eb01226e390f71e WatchSource:0}: Error finding container 05f79ec45d5461b74490185c1e53588c7cf3736a8320fb7b8eb01226e390f71e: Status 404 returned error can't find the container with id 05f79ec45d5461b74490185c1e53588c7cf3736a8320fb7b8eb01226e390f71e Oct 29 22:50:43.816381 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:43.816312 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" event={"ID":"5f7503b3-a2b7-4795-b78a-9f18d00b5f2f","Type":"ContainerStarted","Data":"05f79ec45d5461b74490185c1e53588c7cf3736a8320fb7b8eb01226e390f71e"} Oct 29 22:50:45.827224 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:45.827165 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" event={"ID":"5f7503b3-a2b7-4795-b78a-9f18d00b5f2f","Type":"ContainerStarted","Data":"fe94333d68a821bdc0d07ea64e2f24817e7d53e3881c9f209faa3a5e82db8c75"} Oct 29 22:50:45.877890 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:45.877832 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" podStartSLOduration=2.21386062 podStartE2EDuration="3.877818122s" totalImagesPullingTime="1.663957502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:45.875972571 +0000 UTC m=+492.527471105" watchObservedRunningTime="2026-10-29 22:50:45.877818122 +0000 UTC m=+492.529316651" Oct 29 22:50:46.076765 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:46.076678 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:46.089779 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:46.089705 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:46.127096 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:46.127066 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:46.194774 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:46.194743 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:50:46.842734 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:46.842702 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" Oct 29 22:50:49.084462 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:49.084429 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:49.097039 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:49.097007 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:49.860291 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:49.860262 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" Oct 29 22:50:53.127126 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:53.127092 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:53.139844 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:53.139820 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:53.875499 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:53.875467 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" Oct 29 22:50:54.907897 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:50:54.907859 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:51:03.180825 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.180788 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l"] Oct 29 22:51:03.184766 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.184740 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.203338 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.203315 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h76hm\"" Oct 29 22:51:03.221851 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.221814 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l"] Oct 29 22:51:03.278028 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.277989 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.278224 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.278047 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.278224 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.278124 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.278224 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.278164 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.278325 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.278275 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.278325 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.278316 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkq6f\" (UniqueName: \"kubernetes.io/projected/196b11f8-6a98-45c1-800b-8de400a902bf-kube-api-access-kkq6f\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.278387 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.278342 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196b11f8-6a98-45c1-800b-8de400a902bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.278387 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.278371 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196b11f8-6a98-45c1-800b-8de400a902bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.278449 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.278430 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196b11f8-6a98-45c1-800b-8de400a902bf-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.379532 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.379497 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.379692 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.379557 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.379692 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.379598 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkq6f\" (UniqueName: \"kubernetes.io/projected/196b11f8-6a98-45c1-800b-8de400a902bf-kube-api-access-kkq6f\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.379692 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.379624 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196b11f8-6a98-45c1-800b-8de400a902bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.379692 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.379649 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196b11f8-6a98-45c1-800b-8de400a902bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.379692 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.379677 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196b11f8-6a98-45c1-800b-8de400a902bf-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.379957 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.379710 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.379957 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.379759 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.379957 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.379783 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.380090 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.380024 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.380090 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.380054 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.380226 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.380139 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.380369 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.380347 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.380455 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.380438 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196b11f8-6a98-45c1-800b-8de400a902bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.381984 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.381960 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.382254 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.382235 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196b11f8-6a98-45c1-800b-8de400a902bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.405287 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.405263 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkq6f\" (UniqueName: \"kubernetes.io/projected/196b11f8-6a98-45c1-800b-8de400a902bf-kube-api-access-kkq6f\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.405380 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.405300 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196b11f8-6a98-45c1-800b-8de400a902bf-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-nlz8l\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.497773 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.497700 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:03.812455 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.812425 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l"] Oct 29 22:51:03.813623 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:51:03.813594 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod196b11f8_6a98_45c1_800b_8de400a902bf.slice/crio-35632aece0edf259d373b54c1f0a2c932a7ac65c639c55a4af42d4c9bc91bf04 WatchSource:0}: Error finding container 35632aece0edf259d373b54c1f0a2c932a7ac65c639c55a4af42d4c9bc91bf04: Status 404 returned error can't find the container with id 35632aece0edf259d373b54c1f0a2c932a7ac65c639c55a4af42d4c9bc91bf04 Oct 29 22:51:03.815791 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.815758 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:03.815850 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.815824 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:03.815885 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.815856 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:03.915521 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:51:03.915478 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:51:03.916975 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.916948 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" event={"ID":"196b11f8-6a98-45c1-800b-8de400a902bf","Type":"ContainerStarted","Data":"a03b2f8f98c2fa185501363e63898766a3f91276265c79d1f983e000d7731aba"} Oct 29 22:51:03.917078 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.916998 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" event={"ID":"196b11f8-6a98-45c1-800b-8de400a902bf","Type":"ContainerStarted","Data":"35632aece0edf259d373b54c1f0a2c932a7ac65c639c55a4af42d4c9bc91bf04"} Oct 29 22:51:03.969293 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:03.969238 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" podStartSLOduration=0.969222908 podStartE2EDuration="969.222908ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:03.967870236 +0000 UTC m=+510.619368768" watchObservedRunningTime="2026-10-29 22:51:03.969222908 +0000 UTC m=+510.620721439" Oct 29 22:51:04.498373 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:04.498332 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:04.500006 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:04.499983 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-nlz8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 29 22:51:04.500085 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:04.500030 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" podUID="196b11f8-6a98-45c1-800b-8de400a902bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 29 22:51:05.498670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:05.498639 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-nlz8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 29 22:51:05.499047 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:05.498686 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" podUID="196b11f8-6a98-45c1-800b-8de400a902bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 29 22:51:06.498829 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:06.498794 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-nlz8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 29 22:51:06.499307 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:06.498840 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" podUID="196b11f8-6a98-45c1-800b-8de400a902bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 29 22:51:07.498432 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:07.498396 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-nlz8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 29 22:51:07.498608 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:07.498451 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" podUID="196b11f8-6a98-45c1-800b-8de400a902bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 29 22:51:08.498497 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:08.498461 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-nlz8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 29 22:51:08.498903 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:08.498519 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" podUID="196b11f8-6a98-45c1-800b-8de400a902bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 29 22:51:09.498811 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:09.498778 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-nlz8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 29 22:51:09.499250 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:09.498829 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" podUID="196b11f8-6a98-45c1-800b-8de400a902bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 29 22:51:10.498863 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:10.498832 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-nlz8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 29 22:51:10.499275 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:10.498883 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" podUID="196b11f8-6a98-45c1-800b-8de400a902bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 29 22:51:12.125467 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:12.125431 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:12.125900 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:12.125871 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:12.126676 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:12.126657 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 22:51:18.198339 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.198305 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j"] Oct 29 22:51:18.201890 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.201864 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.214381 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.214349 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd"] Oct 29 22:51:18.217715 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.217696 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.229777 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.229751 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j"] Oct 29 22:51:18.239141 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.239115 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.239308 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.239161 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.239308 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.239200 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.239308 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.239237 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.239308 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.239276 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq4dc\" (UniqueName: \"kubernetes.io/projected/afeb2ed0-d3ab-4607-91c1-f97eb0662229-kube-api-access-nq4dc\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.239486 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.239339 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.239545 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.239507 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.239545 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.239533 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.239652 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.239551 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.239729 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.239707 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.239795 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.239745 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.239795 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.239770 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.239918 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.239868 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbz9s\" (UniqueName: \"kubernetes.io/projected/227efff8-cd76-476b-baa6-045e35ab4d4f-kube-api-access-vbz9s\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.239983 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.239915 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/227efff8-cd76-476b-baa6-045e35ab4d4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.239983 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.239957 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.240098 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.239992 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.240098 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.240021 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.240098 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.240050 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.245136 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.245110 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd"] Oct 29 22:51:18.340730 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.340702 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.340936 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.340739 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.340936 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.340758 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.340936 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.340782 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.340936 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.340801 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq4dc\" (UniqueName: \"kubernetes.io/projected/afeb2ed0-d3ab-4607-91c1-f97eb0662229-kube-api-access-nq4dc\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.340936 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.340821 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.340936 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.340854 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.340936 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.340887 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.341341 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.340964 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.341341 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.341209 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.341341 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.341268 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.341341 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.341293 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.341341 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.341333 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.341601 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.341345 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.341601 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.341401 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.341601 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.341430 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.341601 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.341434 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbz9s\" (UniqueName: \"kubernetes.io/projected/227efff8-cd76-476b-baa6-045e35ab4d4f-kube-api-access-vbz9s\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.341601 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.341473 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/227efff8-cd76-476b-baa6-045e35ab4d4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.341854 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.341620 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.341854 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.341644 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.341854 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.341667 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.341854 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.341703 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.341854 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.341729 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.341854 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.341759 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.342170 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.341943 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.342170 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.342015 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.342170 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.342132 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/227efff8-cd76-476b-baa6-045e35ab4d4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.342345 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.342320 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.343547 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.343523 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.343667 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.343627 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.344351 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.344327 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.344470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.344449 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.366866 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.366837 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbz9s\" (UniqueName: \"kubernetes.io/projected/227efff8-cd76-476b-baa6-045e35ab4d4f-kube-api-access-vbz9s\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.366951 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.366919 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-crrbd\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.367124 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.367103 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.367161 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.367113 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq4dc\" (UniqueName: \"kubernetes.io/projected/afeb2ed0-d3ab-4607-91c1-f97eb0662229-kube-api-access-nq4dc\") pod \"maas-default-gateway-openshift-default-55fbd65575-88j5j\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.516297 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.516211 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:18.531634 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.531608 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:18.842291 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.842264 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j"] Oct 29 22:51:18.845763 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.845633 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd"] Oct 29 22:51:18.846152 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:51:18.846119 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227efff8_cd76_476b_baa6_045e35ab4d4f.slice/crio-26e917d3f668b8db76eb2b4b828730347d9c293afd3b683b932ffd4b28e8f032 WatchSource:0}: Error finding container 26e917d3f668b8db76eb2b4b828730347d9c293afd3b683b932ffd4b28e8f032: Status 404 returned error can't find the container with id 26e917d3f668b8db76eb2b4b828730347d9c293afd3b683b932ffd4b28e8f032 Oct 29 22:51:18.846485 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.846447 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:18.846596 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.846529 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:18.846596 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.846566 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:18.848277 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.848250 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:18.848389 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.848315 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:18.848389 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:18.848346 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:19.019925 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:19.019889 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" event={"ID":"227efff8-cd76-476b-baa6-045e35ab4d4f","Type":"ContainerStarted","Data":"62661df807d64166aab9fa375ac64c909da5e669aef0ed4608277ba2c97dde20"} Oct 29 22:51:19.020114 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:19.019953 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" event={"ID":"227efff8-cd76-476b-baa6-045e35ab4d4f","Type":"ContainerStarted","Data":"26e917d3f668b8db76eb2b4b828730347d9c293afd3b683b932ffd4b28e8f032"} Oct 29 22:51:19.021754 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:19.021729 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" event={"ID":"afeb2ed0-d3ab-4607-91c1-f97eb0662229","Type":"ContainerStarted","Data":"39be18009970a45af27112fb42775b844e41afabb2858c74ca5f3bff170c2931"} Oct 29 22:51:19.021887 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:19.021773 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" event={"ID":"afeb2ed0-d3ab-4607-91c1-f97eb0662229","Type":"ContainerStarted","Data":"4cd04462a694343e7386f9ac3bcff08641ad4fa12eb402938f3941b56b1adcdc"} Oct 29 22:51:19.072129 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:19.072065 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" podStartSLOduration=1.07204693 podStartE2EDuration="1.07204693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:19.069431763 +0000 UTC m=+525.720930295" watchObservedRunningTime="2026-10-29 22:51:19.07204693 +0000 UTC m=+525.723545464" Oct 29 22:51:19.122422 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:19.122321 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" podStartSLOduration=1.122305647 podStartE2EDuration="1.122305647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:19.12035633 +0000 UTC m=+525.771854860" watchObservedRunningTime="2026-10-29 22:51:19.122305647 +0000 UTC m=+525.773804176" Oct 29 22:51:19.517011 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:19.516979 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:19.518498 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:19.518471 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-88j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 29 22:51:19.518622 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:19.518518 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" podUID="afeb2ed0-d3ab-4607-91c1-f97eb0662229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 29 22:51:19.547211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:19.544841 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:19.547211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:19.545022 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-crrbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 29 22:51:19.547211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:19.545069 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" podUID="227efff8-cd76-476b-baa6-045e35ab4d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 29 22:51:20.517265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:20.517228 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-88j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 29 22:51:20.517656 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:20.517287 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" podUID="afeb2ed0-d3ab-4607-91c1-f97eb0662229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 29 22:51:20.542210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:20.542112 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-crrbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 29 22:51:20.542210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:20.542172 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" podUID="227efff8-cd76-476b-baa6-045e35ab4d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 29 22:51:21.517611 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:21.517573 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-88j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 29 22:51:21.518021 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:21.517625 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" podUID="afeb2ed0-d3ab-4607-91c1-f97eb0662229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 29 22:51:21.544211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:21.543943 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-crrbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 29 22:51:21.544211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:21.544002 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" podUID="227efff8-cd76-476b-baa6-045e35ab4d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 29 22:51:22.516716 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:22.516684 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-88j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 29 22:51:22.516887 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:22.516730 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" podUID="afeb2ed0-d3ab-4607-91c1-f97eb0662229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 29 22:51:22.555212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:22.552576 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-crrbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 29 22:51:22.555212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:22.552634 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" podUID="227efff8-cd76-476b-baa6-045e35ab4d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 29 22:51:23.517292 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:23.517258 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-88j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 29 22:51:23.517520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:23.517489 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" podUID="afeb2ed0-d3ab-4607-91c1-f97eb0662229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 29 22:51:23.537534 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:23.537500 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-crrbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 29 22:51:23.538887 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:23.538856 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" podUID="227efff8-cd76-476b-baa6-045e35ab4d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 29 22:51:24.516928 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:24.516880 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-88j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 29 22:51:24.517366 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:24.516946 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" podUID="afeb2ed0-d3ab-4607-91c1-f97eb0662229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 29 22:51:24.538210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:24.534544 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-crrbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 29 22:51:24.538210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:24.534604 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" podUID="227efff8-cd76-476b-baa6-045e35ab4d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 29 22:51:25.516650 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:25.516600 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-88j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 29 22:51:25.516849 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:25.516669 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" podUID="afeb2ed0-d3ab-4607-91c1-f97eb0662229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 29 22:51:25.532001 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:25.531967 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-crrbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 29 22:51:25.532477 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:25.532037 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" podUID="227efff8-cd76-476b-baa6-045e35ab4d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 29 22:51:27.143505 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:27.143474 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:27.144582 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:27.144561 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:27.145172 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:27.145153 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 22:51:27.165298 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:27.165277 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:27.165572 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:27.165550 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:27.166420 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:27.166398 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 22:51:33.912441 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:51:33.912402 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:51:35.152116 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.152083 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd"] Oct 29 22:51:35.156845 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.156817 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.191506 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.191474 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd"] Oct 29 22:51:35.200506 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.200444 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5db969e0-openshift-default-dockercfg-7cmc2\"" Oct 29 22:51:35.226352 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.226320 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-data\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.226504 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.226361 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-envoy\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.226504 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.226396 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-token\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.226616 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.226509 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-workload-socket\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.226616 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.226547 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-workload-certs\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.226616 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.226567 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78vpb\" (UniqueName: \"kubernetes.io/projected/e8e52a93-8d37-4724-9894-43ad3a4c88b6-kube-api-access-78vpb\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.226616 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.226599 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-podinfo\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.227003 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.226981 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-credential-socket\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.227064 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.227038 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istiod-ca-cert\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.328237 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.328177 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-workload-socket\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.328237 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.328243 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-workload-certs\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.328472 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.328269 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78vpb\" (UniqueName: \"kubernetes.io/projected/e8e52a93-8d37-4724-9894-43ad3a4c88b6-kube-api-access-78vpb\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.328472 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.328306 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-podinfo\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.328472 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.328396 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-credential-socket\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.328472 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.328442 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istiod-ca-cert\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.328662 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.328489 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-data\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.328662 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.328524 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-envoy\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.328662 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.328562 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-token\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.330626 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.329493 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-workload-socket\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.331621 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.331405 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-workload-certs\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.331621 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.331514 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istiod-ca-cert\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.331782 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.331651 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-data\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.331842 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.331815 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-credential-socket\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.335099 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.334288 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-podinfo\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.335099 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.334300 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-envoy\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.354570 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.354547 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-token\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.357498 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.357478 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78vpb\" (UniqueName: \"kubernetes.io/projected/e8e52a93-8d37-4724-9894-43ad3a4c88b6-kube-api-access-78vpb\") pod \"e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.472532 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.472495 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:35.486779 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.485094 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5"] Oct 29 22:51:35.489577 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.489553 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.520320 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.517663 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8042816e-openshift-default-dockercfg-brxjw\"" Oct 29 22:51:35.534026 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.533991 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5"] Oct 29 22:51:35.634962 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.634925 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-podinfo\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.635112 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.634995 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9rsj\" (UniqueName: \"kubernetes.io/projected/315a7746-c6a4-4565-82e0-c49ab6f02a69-kube-api-access-w9rsj\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.635112 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.635048 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-envoy\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.635112 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.635091 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-data\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.635257 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.635135 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-credential-socket\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.635257 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.635229 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-workload-socket\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.635324 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.635261 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/315a7746-c6a4-4565-82e0-c49ab6f02a69-istiod-ca-cert\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.635376 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.635364 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-workload-certs\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.635500 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.635400 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-token\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.743830 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.743737 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rsj\" (UniqueName: \"kubernetes.io/projected/315a7746-c6a4-4565-82e0-c49ab6f02a69-kube-api-access-w9rsj\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.743830 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.743798 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-envoy\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.744091 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.743836 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-data\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.744091 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.743884 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-credential-socket\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.744091 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.743916 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-workload-socket\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.744091 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.743941 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/315a7746-c6a4-4565-82e0-c49ab6f02a69-istiod-ca-cert\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.744091 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.744029 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-workload-certs\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.744091 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.744079 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-token\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.744408 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.744118 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-podinfo\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.747639 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.747009 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-credential-socket\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.750644 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.749599 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/315a7746-c6a4-4565-82e0-c49ab6f02a69-istiod-ca-cert\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.750644 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.749855 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-workload-socket\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.750644 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.750100 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-workload-certs\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.750644 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.750364 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-data\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.751852 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.751821 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-podinfo\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.757394 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.757348 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-envoy\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.785456 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.785359 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9rsj\" (UniqueName: \"kubernetes.io/projected/315a7746-c6a4-4565-82e0-c49ab6f02a69-kube-api-access-w9rsj\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.795208 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.795069 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-token\") pod \"e2e-ait-8042816e-openshift-default-6986659497-b5fk5\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.834953 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.834917 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:35.842691 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.842659 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd"] Oct 29 22:51:35.845910 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:51:35.845880 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e52a93_8d37_4724_9894_43ad3a4c88b6.slice/crio-c6315a3e8d5c9cf4673813edfd59df26af2019247baa4ca1b449815f9d83b999 WatchSource:0}: Error finding container c6315a3e8d5c9cf4673813edfd59df26af2019247baa4ca1b449815f9d83b999: Status 404 returned error can't find the container with id c6315a3e8d5c9cf4673813edfd59df26af2019247baa4ca1b449815f9d83b999 Oct 29 22:51:35.849958 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.849923 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:35.850028 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.849990 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:35.850028 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.850021 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:35.894636 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.893551 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m"] Oct 29 22:51:35.899086 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.899063 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:35.922589 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.922561 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m"] Oct 29 22:51:35.922696 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:35.922661 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-2bef9ca4-gw-openshift-default-dockercfg-jg6c5\"" Oct 29 22:51:36.048517 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.047942 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-workload-certs\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.048517 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.048080 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e48474a0-2e27-4cbc-854b-382285cd2745-istio-podinfo\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.048517 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.048221 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-istio-envoy\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.048517 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.048304 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e48474a0-2e27-4cbc-854b-382285cd2745-istiod-ca-cert\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.048517 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.048333 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e48474a0-2e27-4cbc-854b-382285cd2745-istio-token\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.048517 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.048407 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw6tr\" (UniqueName: \"kubernetes.io/projected/e48474a0-2e27-4cbc-854b-382285cd2745-kube-api-access-zw6tr\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.048517 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.048486 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-workload-socket\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.048983 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.048527 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-credential-socket\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.048983 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.048560 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-istio-data\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.149334 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.149293 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-istio-data\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.149515 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.149353 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-workload-certs\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.149515 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.149397 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e48474a0-2e27-4cbc-854b-382285cd2745-istio-podinfo\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.149515 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.149437 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-istio-envoy\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.149515 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.149507 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e48474a0-2e27-4cbc-854b-382285cd2745-istiod-ca-cert\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.149736 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.149534 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e48474a0-2e27-4cbc-854b-382285cd2745-istio-token\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.149736 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.149564 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw6tr\" (UniqueName: \"kubernetes.io/projected/e48474a0-2e27-4cbc-854b-382285cd2745-kube-api-access-zw6tr\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.149736 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.149604 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-workload-socket\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.149736 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.149643 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-credential-socket\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.149935 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.149819 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-workload-certs\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.149935 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.149868 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-istio-data\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.150042 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.149951 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-credential-socket\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.150412 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.150387 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-workload-socket\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.151430 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.150957 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e48474a0-2e27-4cbc-854b-382285cd2745-istiod-ca-cert\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.152133 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.152112 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-istio-envoy\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.152713 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.152526 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e48474a0-2e27-4cbc-854b-382285cd2745-istio-podinfo\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.156321 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.156284 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" event={"ID":"e8e52a93-8d37-4724-9894-43ad3a4c88b6","Type":"ContainerStarted","Data":"811e910b203891e490acae4927c6127299e39874258fe63ebb8e7aed50ff8798"} Oct 29 22:51:36.156433 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.156350 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" event={"ID":"e8e52a93-8d37-4724-9894-43ad3a4c88b6","Type":"ContainerStarted","Data":"c6315a3e8d5c9cf4673813edfd59df26af2019247baa4ca1b449815f9d83b999"} Oct 29 22:51:36.170794 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.170768 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5"] Oct 29 22:51:36.172567 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:51:36.172537 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod315a7746_c6a4_4565_82e0_c49ab6f02a69.slice/crio-895a8b31cc6c44f8c542510338f9eb2604cdc44f063e5a91052209e66a032cce WatchSource:0}: Error finding container 895a8b31cc6c44f8c542510338f9eb2604cdc44f063e5a91052209e66a032cce: Status 404 returned error can't find the container with id 895a8b31cc6c44f8c542510338f9eb2604cdc44f063e5a91052209e66a032cce Oct 29 22:51:36.174778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.174743 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:36.174869 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.174811 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:36.174869 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.174854 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:36.178110 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.177570 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e48474a0-2e27-4cbc-854b-382285cd2745-istio-token\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.182435 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.182412 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw6tr\" (UniqueName: \"kubernetes.io/projected/e48474a0-2e27-4cbc-854b-382285cd2745-kube-api-access-zw6tr\") pod \"e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.213875 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.213824 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" podStartSLOduration=1.213810245 podStartE2EDuration="1.213810245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:36.211341533 +0000 UTC m=+542.862840066" watchObservedRunningTime="2026-10-29 22:51:36.213810245 +0000 UTC m=+542.865308775" Oct 29 22:51:36.225334 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.225308 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:36.473478 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.473436 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:36.478603 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.478579 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:36.533285 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.533250 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m"] Oct 29 22:51:36.534455 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:51:36.534430 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48474a0_2e27_4cbc_854b_382285cd2745.slice/crio-c912bc94d16297e328a4103e6f6fb7b03c4547f84275fe49c448301d64acf143 WatchSource:0}: Error finding container c912bc94d16297e328a4103e6f6fb7b03c4547f84275fe49c448301d64acf143: Status 404 returned error can't find the container with id c912bc94d16297e328a4103e6f6fb7b03c4547f84275fe49c448301d64acf143 Oct 29 22:51:36.537153 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.537119 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:36.537259 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.537182 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:36.537259 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:36.537233 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:37.162803 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:37.162760 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" event={"ID":"315a7746-c6a4-4565-82e0-c49ab6f02a69","Type":"ContainerStarted","Data":"ac7288d722d07173cdd3a76c508ba7d15308e9e7fdf37123bd6d9d7a2fee3c9e"} Oct 29 22:51:37.163259 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:37.162824 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" event={"ID":"315a7746-c6a4-4565-82e0-c49ab6f02a69","Type":"ContainerStarted","Data":"895a8b31cc6c44f8c542510338f9eb2604cdc44f063e5a91052209e66a032cce"} Oct 29 22:51:37.164528 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:37.164504 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" event={"ID":"e48474a0-2e27-4cbc-854b-382285cd2745","Type":"ContainerStarted","Data":"666b319657c565508cb82205553e9fe0df27e073ad288c820251fe527f7bbcac"} Oct 29 22:51:37.164649 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:37.164558 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:37.164649 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:37.164585 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" event={"ID":"e48474a0-2e27-4cbc-854b-382285cd2745","Type":"ContainerStarted","Data":"c912bc94d16297e328a4103e6f6fb7b03c4547f84275fe49c448301d64acf143"} Oct 29 22:51:37.165555 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:37.165533 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:37.229220 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:37.226600 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" podStartSLOduration=2.226578081 podStartE2EDuration="2.226578081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:37.224810991 +0000 UTC m=+543.876309522" watchObservedRunningTime="2026-10-29 22:51:37.226578081 +0000 UTC m=+543.878076615" Oct 29 22:51:37.230258 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:37.230224 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:37.245303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:37.245274 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:37.344231 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:37.335676 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" podStartSLOduration=2.335656892 podStartE2EDuration="2.335656892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:37.334339107 +0000 UTC m=+543.985837638" watchObservedRunningTime="2026-10-29 22:51:37.335656892 +0000 UTC m=+543.987155422" Oct 29 22:51:37.835835 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:37.835802 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:37.841159 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:37.841133 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:38.170920 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:38.170837 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:38.171515 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:38.171486 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:38.171644 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:38.171568 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:51:38.171644 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:38.171614 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:51:41.416999 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:41.416965 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm"] Oct 29 22:51:41.421742 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:41.421725 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm" Oct 29 22:51:41.443620 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:41.443586 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm"] Oct 29 22:51:41.455124 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:41.455099 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-e406071d\"" Oct 29 22:51:41.455270 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:41.455251 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-e406071d-dockercfg-p4t4q\"" Oct 29 22:51:41.540958 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:41.540922 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6cxm\" (UniqueName: \"kubernetes.io/projected/da8ef65e-3b78-4373-b94e-7c54b8b7971d-kube-api-access-x6cxm\") pod \"payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm\" (UID: \"da8ef65e-3b78-4373-b94e-7c54b8b7971d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm" Oct 29 22:51:41.541225 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:41.541200 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8ef65e-3b78-4373-b94e-7c54b8b7971d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm\" (UID: \"da8ef65e-3b78-4373-b94e-7c54b8b7971d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm" Oct 29 22:51:41.642708 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:41.642664 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6cxm\" (UniqueName: \"kubernetes.io/projected/da8ef65e-3b78-4373-b94e-7c54b8b7971d-kube-api-access-x6cxm\") pod \"payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm\" (UID: \"da8ef65e-3b78-4373-b94e-7c54b8b7971d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm" Oct 29 22:51:41.642900 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:41.642715 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8ef65e-3b78-4373-b94e-7c54b8b7971d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm\" (UID: \"da8ef65e-3b78-4373-b94e-7c54b8b7971d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm" Oct 29 22:51:41.643878 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:41.643855 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8ef65e-3b78-4373-b94e-7c54b8b7971d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm\" (UID: \"da8ef65e-3b78-4373-b94e-7c54b8b7971d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm" Oct 29 22:51:41.670708 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:41.670637 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6cxm\" (UniqueName: \"kubernetes.io/projected/da8ef65e-3b78-4373-b94e-7c54b8b7971d-kube-api-access-x6cxm\") pod \"payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm\" (UID: \"da8ef65e-3b78-4373-b94e-7c54b8b7971d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm" Oct 29 22:51:41.735082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:41.735046 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm" Oct 29 22:51:42.064880 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:42.064855 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm"] Oct 29 22:51:42.066981 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:51:42.066945 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8ef65e_3b78_4373_b94e_7c54b8b7971d.slice/crio-6a7325493452672067e902797e6729870941ea79956af0eb301fd5bd639a29c5 WatchSource:0}: Error finding container 6a7325493452672067e902797e6729870941ea79956af0eb301fd5bd639a29c5: Status 404 returned error can't find the container with id 6a7325493452672067e902797e6729870941ea79956af0eb301fd5bd639a29c5 Oct 29 22:51:42.217607 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:42.217149 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm" event={"ID":"da8ef65e-3b78-4373-b94e-7c54b8b7971d","Type":"ContainerStarted","Data":"6a7325493452672067e902797e6729870941ea79956af0eb301fd5bd639a29c5"} Oct 29 22:51:42.302098 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:42.302060 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz"] Oct 29 22:51:42.307230 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:42.307201 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz" Oct 29 22:51:42.326143 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:42.326115 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-865d946f\"" Oct 29 22:51:42.329439 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:42.329412 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz"] Oct 29 22:51:42.458413 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:42.458307 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxvmb\" (UniqueName: \"kubernetes.io/projected/316c85fe-2455-4fad-840f-e945ab9c57d7-kube-api-access-qxvmb\") pod \"payload-processing-e2e-ait-865d946f-5f594694bb-wsszz\" (UID: \"316c85fe-2455-4fad-840f-e945ab9c57d7\") " pod="openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz" Oct 29 22:51:42.459728 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:42.459638 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/316c85fe-2455-4fad-840f-e945ab9c57d7-plugins-config-volume\") pod \"payload-processing-e2e-ait-865d946f-5f594694bb-wsszz\" (UID: \"316c85fe-2455-4fad-840f-e945ab9c57d7\") " pod="openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz" Oct 29 22:51:42.566280 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:42.565423 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/316c85fe-2455-4fad-840f-e945ab9c57d7-plugins-config-volume\") pod \"payload-processing-e2e-ait-865d946f-5f594694bb-wsszz\" (UID: \"316c85fe-2455-4fad-840f-e945ab9c57d7\") " pod="openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz" Oct 29 22:51:42.566280 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:42.565612 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxvmb\" (UniqueName: \"kubernetes.io/projected/316c85fe-2455-4fad-840f-e945ab9c57d7-kube-api-access-qxvmb\") pod \"payload-processing-e2e-ait-865d946f-5f594694bb-wsszz\" (UID: \"316c85fe-2455-4fad-840f-e945ab9c57d7\") " pod="openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz" Oct 29 22:51:42.567316 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:42.567254 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/316c85fe-2455-4fad-840f-e945ab9c57d7-plugins-config-volume\") pod \"payload-processing-e2e-ait-865d946f-5f594694bb-wsszz\" (UID: \"316c85fe-2455-4fad-840f-e945ab9c57d7\") " pod="openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz" Oct 29 22:51:42.608733 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:42.608543 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxvmb\" (UniqueName: \"kubernetes.io/projected/316c85fe-2455-4fad-840f-e945ab9c57d7-kube-api-access-qxvmb\") pod \"payload-processing-e2e-ait-865d946f-5f594694bb-wsszz\" (UID: \"316c85fe-2455-4fad-840f-e945ab9c57d7\") " pod="openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz" Oct 29 22:51:42.621451 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:42.621124 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz" Oct 29 22:51:42.804656 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:42.804614 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:51:42.967230 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:42.967200 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz"] Oct 29 22:51:42.973522 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:51:42.973490 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod316c85fe_2455_4fad_840f_e945ab9c57d7.slice/crio-ff331c670a05979f3866a6985156c666a2a48bf05261947aef6748bfb344e3eb WatchSource:0}: Error finding container ff331c670a05979f3866a6985156c666a2a48bf05261947aef6748bfb344e3eb: Status 404 returned error can't find the container with id ff331c670a05979f3866a6985156c666a2a48bf05261947aef6748bfb344e3eb Oct 29 22:51:43.225637 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.225555 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz" event={"ID":"316c85fe-2455-4fad-840f-e945ab9c57d7","Type":"ContainerStarted","Data":"ff331c670a05979f3866a6985156c666a2a48bf05261947aef6748bfb344e3eb"} Oct 29 22:51:43.341155 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.341104 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw"] Oct 29 22:51:43.346854 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.346828 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw" Oct 29 22:51:43.365293 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.365265 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-2bef9ca4\"" Oct 29 22:51:43.368075 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.368042 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw"] Oct 29 22:51:43.379492 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.379461 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-2bef9ca4-dockercfg-rvqrs\"" Oct 29 22:51:43.450775 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.449239 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk"] Oct 29 22:51:43.478099 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.463321 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk"] Oct 29 22:51:43.478099 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.463493 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk" Oct 29 22:51:43.478099 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.475335 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b95244-83c6-4396-aaaf-6399a5e817b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw\" (UID: \"51b95244-83c6-4396-aaaf-6399a5e817b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw" Oct 29 22:51:43.478099 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.475468 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qndxm\" (UniqueName: \"kubernetes.io/projected/51b95244-83c6-4396-aaaf-6399a5e817b3-kube-api-access-qndxm\") pod \"payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw\" (UID: \"51b95244-83c6-4396-aaaf-6399a5e817b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw" Oct 29 22:51:43.578551 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.576513 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qndxm\" (UniqueName: \"kubernetes.io/projected/51b95244-83c6-4396-aaaf-6399a5e817b3-kube-api-access-qndxm\") pod \"payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw\" (UID: \"51b95244-83c6-4396-aaaf-6399a5e817b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw" Oct 29 22:51:43.578551 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.576606 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd6s8\" (UniqueName: \"kubernetes.io/projected/7dc73919-716d-4f42-91c4-33c9e4f30f07-kube-api-access-dd6s8\") pod \"payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk\" (UID: \"7dc73919-716d-4f42-91c4-33c9e4f30f07\") " pod="openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk" Oct 29 22:51:43.578551 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.576680 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc73919-716d-4f42-91c4-33c9e4f30f07-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk\" (UID: \"7dc73919-716d-4f42-91c4-33c9e4f30f07\") " pod="openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk" Oct 29 22:51:43.578551 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.576705 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b95244-83c6-4396-aaaf-6399a5e817b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw\" (UID: \"51b95244-83c6-4396-aaaf-6399a5e817b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw" Oct 29 22:51:43.578551 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.577535 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b95244-83c6-4396-aaaf-6399a5e817b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw\" (UID: \"51b95244-83c6-4396-aaaf-6399a5e817b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw" Oct 29 22:51:43.615546 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.610285 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qndxm\" (UniqueName: \"kubernetes.io/projected/51b95244-83c6-4396-aaaf-6399a5e817b3-kube-api-access-qndxm\") pod \"payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw\" (UID: \"51b95244-83c6-4396-aaaf-6399a5e817b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw" Oct 29 22:51:43.667061 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.666717 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw" Oct 29 22:51:43.677709 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.677673 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd6s8\" (UniqueName: \"kubernetes.io/projected/7dc73919-716d-4f42-91c4-33c9e4f30f07-kube-api-access-dd6s8\") pod \"payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk\" (UID: \"7dc73919-716d-4f42-91c4-33c9e4f30f07\") " pod="openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk" Oct 29 22:51:43.677841 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.677775 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc73919-716d-4f42-91c4-33c9e4f30f07-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk\" (UID: \"7dc73919-716d-4f42-91c4-33c9e4f30f07\") " pod="openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk" Oct 29 22:51:43.679097 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.679065 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc73919-716d-4f42-91c4-33c9e4f30f07-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk\" (UID: \"7dc73919-716d-4f42-91c4-33c9e4f30f07\") " pod="openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk" Oct 29 22:51:43.713379 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.713341 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd6s8\" (UniqueName: \"kubernetes.io/projected/7dc73919-716d-4f42-91c4-33c9e4f30f07-kube-api-access-dd6s8\") pod \"payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk\" (UID: \"7dc73919-716d-4f42-91c4-33c9e4f30f07\") " pod="openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk" Oct 29 22:51:43.831215 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:43.811722 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk" Oct 29 22:51:44.064014 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:44.062734 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw"] Oct 29 22:51:44.072708 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:51:44.072613 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b95244_83c6_4396_aaaf_6399a5e817b3.slice/crio-d2297efb8e962c29c7d23f2e545fca94fc1fed0396d928fe8fb1ca3016d30e91 WatchSource:0}: Error finding container d2297efb8e962c29c7d23f2e545fca94fc1fed0396d928fe8fb1ca3016d30e91: Status 404 returned error can't find the container with id d2297efb8e962c29c7d23f2e545fca94fc1fed0396d928fe8fb1ca3016d30e91 Oct 29 22:51:44.180963 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:44.180931 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk"] Oct 29 22:51:44.183221 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:51:44.183195 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc73919_716d_4f42_91c4_33c9e4f30f07.slice/crio-9081c68715bba436c3785ec3ef8dcf99bb2084c36d701597071722927430b6bf WatchSource:0}: Error finding container 9081c68715bba436c3785ec3ef8dcf99bb2084c36d701597071722927430b6bf: Status 404 returned error can't find the container with id 9081c68715bba436c3785ec3ef8dcf99bb2084c36d701597071722927430b6bf Oct 29 22:51:44.265369 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:44.265267 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk" event={"ID":"7dc73919-716d-4f42-91c4-33c9e4f30f07","Type":"ContainerStarted","Data":"9081c68715bba436c3785ec3ef8dcf99bb2084c36d701597071722927430b6bf"} Oct 29 22:51:44.267354 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:44.267324 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw" event={"ID":"51b95244-83c6-4396-aaaf-6399a5e817b3","Type":"ContainerStarted","Data":"d2297efb8e962c29c7d23f2e545fca94fc1fed0396d928fe8fb1ca3016d30e91"} Oct 29 22:51:46.250216 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:46.249665 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9"] Oct 29 22:51:46.260667 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:46.260638 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9" Oct 29 22:51:46.275350 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:46.274633 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9"] Oct 29 22:51:46.284756 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:46.284723 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8042816e\"" Oct 29 22:51:46.390492 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:46.385623 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lk4q\" (UniqueName: \"kubernetes.io/projected/4dc36c8e-d2aa-41d8-a72b-237d0d3f530c-kube-api-access-9lk4q\") pod \"payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9\" (UID: \"4dc36c8e-d2aa-41d8-a72b-237d0d3f530c\") " pod="openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9" Oct 29 22:51:46.390492 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:46.385726 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc36c8e-d2aa-41d8-a72b-237d0d3f530c-plugins-config-volume\") pod \"payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9\" (UID: \"4dc36c8e-d2aa-41d8-a72b-237d0d3f530c\") " pod="openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9" Oct 29 22:51:46.486776 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:46.486744 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lk4q\" (UniqueName: \"kubernetes.io/projected/4dc36c8e-d2aa-41d8-a72b-237d0d3f530c-kube-api-access-9lk4q\") pod \"payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9\" (UID: \"4dc36c8e-d2aa-41d8-a72b-237d0d3f530c\") " pod="openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9" Oct 29 22:51:46.486976 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:46.486820 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc36c8e-d2aa-41d8-a72b-237d0d3f530c-plugins-config-volume\") pod \"payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9\" (UID: \"4dc36c8e-d2aa-41d8-a72b-237d0d3f530c\") " pod="openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9" Oct 29 22:51:46.487637 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:46.487610 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc36c8e-d2aa-41d8-a72b-237d0d3f530c-plugins-config-volume\") pod \"payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9\" (UID: \"4dc36c8e-d2aa-41d8-a72b-237d0d3f530c\") " pod="openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9" Oct 29 22:51:46.514807 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:46.514731 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lk4q\" (UniqueName: \"kubernetes.io/projected/4dc36c8e-d2aa-41d8-a72b-237d0d3f530c-kube-api-access-9lk4q\") pod \"payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9\" (UID: \"4dc36c8e-d2aa-41d8-a72b-237d0d3f530c\") " pod="openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9" Oct 29 22:51:46.582016 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:46.581977 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9" Oct 29 22:51:46.933465 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:46.931163 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9"] Oct 29 22:51:46.950171 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:51:46.950133 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc36c8e_d2aa_41d8_a72b_237d0d3f530c.slice/crio-c8e6c58825cd8fd0dd7278afd16657ba53b21a9faf920048b065a02fb445afc7 WatchSource:0}: Error finding container c8e6c58825cd8fd0dd7278afd16657ba53b21a9faf920048b065a02fb445afc7: Status 404 returned error can't find the container with id c8e6c58825cd8fd0dd7278afd16657ba53b21a9faf920048b065a02fb445afc7 Oct 29 22:51:47.297209 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:47.297125 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9" event={"ID":"4dc36c8e-d2aa-41d8-a72b-237d0d3f530c","Type":"ContainerStarted","Data":"c8e6c58825cd8fd0dd7278afd16657ba53b21a9faf920048b065a02fb445afc7"} Oct 29 22:51:47.900369 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:47.900010 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95"] Oct 29 22:51:47.911807 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:47.911781 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95" Oct 29 22:51:47.940095 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:47.940063 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-e57216-dockercfg-ctfbl\"" Oct 29 22:51:47.966435 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:47.966200 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86rf8\" (UniqueName: \"kubernetes.io/projected/ed5ee942-63e4-448f-b180-6790f887ffa6-kube-api-access-86rf8\") pod \"payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95\" (UID: \"ed5ee942-63e4-448f-b180-6790f887ffa6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95" Oct 29 22:51:47.966435 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:47.966283 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed5ee942-63e4-448f-b180-6790f887ffa6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95\" (UID: \"ed5ee942-63e4-448f-b180-6790f887ffa6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95" Oct 29 22:51:47.968443 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:47.968404 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-e57216\"" Oct 29 22:51:47.988313 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:47.979386 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95"] Oct 29 22:51:48.072838 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:48.072800 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86rf8\" (UniqueName: \"kubernetes.io/projected/ed5ee942-63e4-448f-b180-6790f887ffa6-kube-api-access-86rf8\") pod \"payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95\" (UID: \"ed5ee942-63e4-448f-b180-6790f887ffa6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95" Oct 29 22:51:48.072983 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:48.072871 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed5ee942-63e4-448f-b180-6790f887ffa6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95\" (UID: \"ed5ee942-63e4-448f-b180-6790f887ffa6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95" Oct 29 22:51:48.073838 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:48.073767 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed5ee942-63e4-448f-b180-6790f887ffa6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95\" (UID: \"ed5ee942-63e4-448f-b180-6790f887ffa6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95" Oct 29 22:51:48.100254 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:48.100224 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86rf8\" (UniqueName: \"kubernetes.io/projected/ed5ee942-63e4-448f-b180-6790f887ffa6-kube-api-access-86rf8\") pod \"payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95\" (UID: \"ed5ee942-63e4-448f-b180-6790f887ffa6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95" Oct 29 22:51:48.306358 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:48.305550 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk" event={"ID":"7dc73919-716d-4f42-91c4-33c9e4f30f07","Type":"ContainerStarted","Data":"34ba12aac64e08dca7241b73c43da8f0fed59318e44f077022d481eacf3e5f33"} Oct 29 22:51:48.306358 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:48.306311 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk" Oct 29 22:51:48.309473 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:48.309144 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm" event={"ID":"da8ef65e-3b78-4373-b94e-7c54b8b7971d","Type":"ContainerStarted","Data":"8168a8c9353d846cc2580b7c55badf493629b2a33ee5773e26f8b52f0e6559ee"} Oct 29 22:51:48.309473 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:48.309440 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm" Oct 29 22:51:48.311053 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:48.310628 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9" event={"ID":"4dc36c8e-d2aa-41d8-a72b-237d0d3f530c","Type":"ContainerStarted","Data":"105ee2e5d84832f0c03c8c6de34bd9974a7182f1394c1538e3c741958ee95b6d"} Oct 29 22:51:48.311053 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:48.310755 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9" Oct 29 22:51:48.313234 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:48.313215 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw" event={"ID":"51b95244-83c6-4396-aaaf-6399a5e817b3","Type":"ContainerStarted","Data":"5d58c0ad1585ad92060a55c498af716ba3f93d5595fdb7c077c02fe10581a7d0"} Oct 29 22:51:48.313343 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:48.313333 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw" Oct 29 22:51:48.315881 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:48.315863 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95" Oct 29 22:51:48.358778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:48.358721 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk" podStartSLOduration=1.549449957 podStartE2EDuration="5.358708206s" totalImagesPullingTime="3.809258249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:48.35827018 +0000 UTC m=+555.009768711" watchObservedRunningTime="2026-10-29 22:51:48.358708206 +0000 UTC m=+555.010206727" Oct 29 22:51:48.420230 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:48.420121 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9" podStartSLOduration=1.5585176939999998 podStartE2EDuration="2.420094961s" totalImagesPullingTime="861.577267ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:48.41946758 +0000 UTC m=+555.070966114" watchObservedRunningTime="2026-10-29 22:51:48.420094961 +0000 UTC m=+555.071593493" Oct 29 22:51:48.567402 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:48.566975 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm" podStartSLOduration=2.136371252 podStartE2EDuration="7.566952603s" totalImagesPullingTime="5.430581351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:48.497091628 +0000 UTC m=+555.148590160" watchObservedRunningTime="2026-10-29 22:51:48.566952603 +0000 UTC m=+555.218451136" Oct 29 22:51:48.858368 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:48.856654 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw" podStartSLOduration=2.241577728 podStartE2EDuration="5.856632513s" totalImagesPullingTime="3.615054785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:48.571065254 +0000 UTC m=+555.222563785" watchObservedRunningTime="2026-10-29 22:51:48.856632513 +0000 UTC m=+555.508131044" Oct 29 22:51:48.858368 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:48.857871 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95"] Oct 29 22:51:49.066576 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:49.066537 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:51:49.364782 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:49.364742 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz" event={"ID":"316c85fe-2455-4fad-840f-e945ab9c57d7","Type":"ContainerStarted","Data":"35ed0b89ef3e0299942b82b639cc44bab37c2cfbe46278bd8a225d0793d3df01"} Oct 29 22:51:49.365738 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:49.365716 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz" Oct 29 22:51:49.368765 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:49.368740 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95" event={"ID":"ed5ee942-63e4-448f-b180-6790f887ffa6","Type":"ContainerStarted","Data":"e0bd03ebbd7f56ca74c74dad59864fa3ce069df4080483003ab3000f87b69032"} Oct 29 22:51:49.368811 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:49.368793 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95" Oct 29 22:51:49.368841 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:49.368817 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95" event={"ID":"ed5ee942-63e4-448f-b180-6790f887ffa6","Type":"ContainerStarted","Data":"d183e9ea1c8db2a71da46e621728cce1ddfb6a2c38eb96dbf773649300691c77"} Oct 29 22:51:49.487427 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:49.487332 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz" podStartSLOduration=1.615393828 podStartE2EDuration="7.48731426s" totalImagesPullingTime="5.871920432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:49.468796598 +0000 UTC m=+556.120295129" watchObservedRunningTime="2026-10-29 22:51:49.48731426 +0000 UTC m=+556.138812787" Oct 29 22:51:49.529554 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:49.529498 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95" podStartSLOduration=2.5294852150000002 podStartE2EDuration="2.529485215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:49.527805201 +0000 UTC m=+556.179303736" watchObservedRunningTime="2026-10-29 22:51:49.529485215 +0000 UTC m=+556.180983744" Oct 29 22:51:51.554288 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:51.554253 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh"] Oct 29 22:51:51.560290 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:51.560262 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh" Oct 29 22:51:51.564469 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:51.564428 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037f1359-4def-4ce0-a40f-4db85cb069cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh\" (UID: \"037f1359-4def-4ce0-a40f-4db85cb069cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh" Oct 29 22:51:51.564668 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:51.564534 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjfhv\" (UniqueName: \"kubernetes.io/projected/037f1359-4def-4ce0-a40f-4db85cb069cc-kube-api-access-rjfhv\") pod \"payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh\" (UID: \"037f1359-4def-4ce0-a40f-4db85cb069cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh" Oct 29 22:51:51.580017 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:51.579979 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh"] Oct 29 22:51:51.591922 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:51.591869 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-99dfb0-dockercfg-bsszb\"" Oct 29 22:51:51.592063 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:51.592026 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-99dfb0\"" Oct 29 22:51:51.665669 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:51.665626 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjfhv\" (UniqueName: \"kubernetes.io/projected/037f1359-4def-4ce0-a40f-4db85cb069cc-kube-api-access-rjfhv\") pod \"payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh\" (UID: \"037f1359-4def-4ce0-a40f-4db85cb069cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh" Oct 29 22:51:51.665874 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:51.665753 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037f1359-4def-4ce0-a40f-4db85cb069cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh\" (UID: \"037f1359-4def-4ce0-a40f-4db85cb069cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh" Oct 29 22:51:51.666444 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:51.666421 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037f1359-4def-4ce0-a40f-4db85cb069cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh\" (UID: \"037f1359-4def-4ce0-a40f-4db85cb069cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh" Oct 29 22:51:51.691209 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:51.691158 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjfhv\" (UniqueName: \"kubernetes.io/projected/037f1359-4def-4ce0-a40f-4db85cb069cc-kube-api-access-rjfhv\") pod \"payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh\" (UID: \"037f1359-4def-4ce0-a40f-4db85cb069cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh" Oct 29 22:51:51.874654 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:51.874562 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh" Oct 29 22:51:52.201701 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:52.201669 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh"] Oct 29 22:51:52.208960 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:51:52.208921 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037f1359_4def_4ce0_a40f_4db85cb069cc.slice/crio-0304daca905eec59c755d533b4496cff713af56e6b22e18db2b3e30129963e55 WatchSource:0}: Error finding container 0304daca905eec59c755d533b4496cff713af56e6b22e18db2b3e30129963e55: Status 404 returned error can't find the container with id 0304daca905eec59c755d533b4496cff713af56e6b22e18db2b3e30129963e55 Oct 29 22:51:52.395526 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:52.395485 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh" event={"ID":"037f1359-4def-4ce0-a40f-4db85cb069cc","Type":"ContainerStarted","Data":"0304daca905eec59c755d533b4496cff713af56e6b22e18db2b3e30129963e55"} Oct 29 22:51:53.207387 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:53.207343 5057 patch_prober.go:29] interesting pod/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:51:53.207810 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:53.207411 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" podUID="e8e52a93-8d37-4724-9894-43ad3a4c88b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:51:53.404836 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:53.404569 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh" event={"ID":"037f1359-4def-4ce0-a40f-4db85cb069cc","Type":"ContainerStarted","Data":"7c7d9e7912b07ff09d3e05adebea53d0877a186b4f7a850211fc8f4864893ae5"} Oct 29 22:51:53.404836 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:53.404702 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh" Oct 29 22:51:53.456371 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:53.456300 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh" podStartSLOduration=2.456279812 podStartE2EDuration="2.456279812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:53.455765739 +0000 UTC m=+560.107264270" watchObservedRunningTime="2026-10-29 22:51:53.456279812 +0000 UTC m=+560.107778345" Oct 29 22:51:54.411558 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:54.411482 5057 generic.go:356] "Generic (PLEG): container finished" podID="e8e52a93-8d37-4724-9894-43ad3a4c88b6" containerID="811e910b203891e490acae4927c6127299e39874258fe63ebb8e7aed50ff8798" exitCode=0 Oct 29 22:51:54.411983 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:54.411551 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" event={"ID":"e8e52a93-8d37-4724-9894-43ad3a4c88b6","Type":"ContainerDied","Data":"811e910b203891e490acae4927c6127299e39874258fe63ebb8e7aed50ff8798"} Oct 29 22:51:54.412102 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:54.412087 5057 scope.go:122] "RemoveContainer" containerID="811e910b203891e490acae4927c6127299e39874258fe63ebb8e7aed50ff8798" Oct 29 22:51:54.413360 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:54.413328 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:54.413456 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:54.413400 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:54.413456 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:54.413439 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:51:55.460394 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:55.459392 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" event={"ID":"e8e52a93-8d37-4724-9894-43ad3a4c88b6","Type":"ContainerStarted","Data":"94b574ba71171752787e9faf697605d728602bae34ce7dd2da1ef2efa2ad05c8"} Oct 29 22:51:55.481893 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:55.481856 5057 patch_prober.go:29] interesting pod/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 29 22:51:55.482225 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:55.482159 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" podUID="e8e52a93-8d37-4724-9894-43ad3a4c88b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 29 22:51:55.484946 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:55.484910 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:51:55.899530 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:55.899454 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:51:56.473454 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:56.473413 5057 patch_prober.go:29] interesting pod/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 29 22:51:56.473912 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:56.473499 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" podUID="e8e52a93-8d37-4724-9894-43ad3a4c88b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 29 22:51:57.483556 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:57.483427 5057 patch_prober.go:29] interesting pod/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 29 22:51:57.483556 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:57.483504 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" podUID="e8e52a93-8d37-4724-9894-43ad3a4c88b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 29 22:51:58.473750 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:58.473703 5057 patch_prober.go:29] interesting pod/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 29 22:51:58.473970 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:58.473817 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" podUID="e8e52a93-8d37-4724-9894-43ad3a4c88b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 29 22:51:58.808486 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:58.808410 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8"] Oct 29 22:51:58.815161 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:58.815138 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8" Oct 29 22:51:58.836095 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:58.836058 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8"] Oct 29 22:51:58.850209 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:58.849545 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:51:58.850209 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:58.849623 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:51:58.850209 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:58.849685 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:51:58.850209 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:58.849778 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:51:58.906543 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:58.906495 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8\" (UID: \"983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8" Oct 29 22:51:58.906887 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:58.906867 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btj8l\" (UniqueName: \"kubernetes.io/projected/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-kube-api-access-btj8l\") pod \"maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8\" (UID: \"983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8" Oct 29 22:51:58.907046 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:58.907031 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8\" (UID: \"983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8" Oct 29 22:51:59.008582 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.008545 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8\" (UID: \"983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8" Oct 29 22:51:59.008794 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.008641 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btj8l\" (UniqueName: \"kubernetes.io/projected/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-kube-api-access-btj8l\") pod \"maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8\" (UID: \"983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8" Oct 29 22:51:59.009078 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.009060 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8\" (UID: \"983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8" Oct 29 22:51:59.009233 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.009216 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8\" (UID: \"983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8" Oct 29 22:51:59.011879 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.011861 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8\" (UID: \"983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8" Oct 29 22:51:59.073083 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.072950 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btj8l\" (UniqueName: \"kubernetes.io/projected/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-kube-api-access-btj8l\") pod \"maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8\" (UID: \"983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8" Oct 29 22:51:59.153413 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.153375 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8" Oct 29 22:51:59.267212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.257300 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6"] Oct 29 22:51:59.283150 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.282406 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6" Oct 29 22:51:59.290000 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.289963 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6"] Oct 29 22:51:59.314672 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.313815 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fb57d7e-62d6-473b-b381-179752f820b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6\" (UID: \"5fb57d7e-62d6-473b-b381-179752f820b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6" Oct 29 22:51:59.314672 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.314021 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fb57d7e-62d6-473b-b381-179752f820b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6\" (UID: \"5fb57d7e-62d6-473b-b381-179752f820b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6" Oct 29 22:51:59.314672 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.314084 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwz26\" (UniqueName: \"kubernetes.io/projected/5fb57d7e-62d6-473b-b381-179752f820b8-kube-api-access-lwz26\") pod \"maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6\" (UID: \"5fb57d7e-62d6-473b-b381-179752f820b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6" Oct 29 22:51:59.403232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.403202 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw" Oct 29 22:51:59.403987 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.403485 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9" Oct 29 22:51:59.416515 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.415504 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fb57d7e-62d6-473b-b381-179752f820b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6\" (UID: \"5fb57d7e-62d6-473b-b381-179752f820b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6" Oct 29 22:51:59.416515 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.415548 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwz26\" (UniqueName: \"kubernetes.io/projected/5fb57d7e-62d6-473b-b381-179752f820b8-kube-api-access-lwz26\") pod \"maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6\" (UID: \"5fb57d7e-62d6-473b-b381-179752f820b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6" Oct 29 22:51:59.416515 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.415597 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fb57d7e-62d6-473b-b381-179752f820b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6\" (UID: \"5fb57d7e-62d6-473b-b381-179752f820b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6" Oct 29 22:51:59.416515 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.416110 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fb57d7e-62d6-473b-b381-179752f820b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6\" (UID: \"5fb57d7e-62d6-473b-b381-179752f820b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6" Oct 29 22:51:59.418663 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.418604 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fb57d7e-62d6-473b-b381-179752f820b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6\" (UID: \"5fb57d7e-62d6-473b-b381-179752f820b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6" Oct 29 22:51:59.476212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.475199 5057 patch_prober.go:29] interesting pod/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 29 22:51:59.476212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.475272 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" podUID="e8e52a93-8d37-4724-9894-43ad3a4c88b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 29 22:51:59.506977 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.506938 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwz26\" (UniqueName: \"kubernetes.io/projected/5fb57d7e-62d6-473b-b381-179752f820b8-kube-api-access-lwz26\") pod \"maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6\" (UID: \"5fb57d7e-62d6-473b-b381-179752f820b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6" Oct 29 22:51:59.586147 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.583476 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8"] Oct 29 22:51:59.587423 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:51:59.587394 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983e6ca2_bd70_42bd_9b66_78cdbb0ff8f5.slice/crio-ba38cb0cb6ff3201ef517697a5b916f2398fdd1425747468cde3e004fb9456dd WatchSource:0}: Error finding container ba38cb0cb6ff3201ef517697a5b916f2398fdd1425747468cde3e004fb9456dd: Status 404 returned error can't find the container with id ba38cb0cb6ff3201ef517697a5b916f2398fdd1425747468cde3e004fb9456dd Oct 29 22:51:59.608166 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:51:59.608093 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6" Oct 29 22:52:00.142214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:00.120121 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6"] Oct 29 22:52:00.395557 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:00.394986 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk" Oct 29 22:52:00.395557 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:00.395247 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm" Oct 29 22:52:00.395557 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:00.395366 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95" Oct 29 22:52:00.494340 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:00.494302 5057 patch_prober.go:29] interesting pod/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 29 22:52:00.494546 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:00.494362 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" podUID="e8e52a93-8d37-4724-9894-43ad3a4c88b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 29 22:52:00.520751 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:00.520679 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6" event={"ID":"5fb57d7e-62d6-473b-b381-179752f820b8","Type":"ContainerStarted","Data":"ca57113897700f46964faea7d3b8a4d50f113e9447f04db617072c716eb616dc"} Oct 29 22:52:00.577503 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:00.577412 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8" event={"ID":"983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5","Type":"ContainerStarted","Data":"ba38cb0cb6ff3201ef517697a5b916f2398fdd1425747468cde3e004fb9456dd"} Oct 29 22:52:01.389684 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:01.389623 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz" Oct 29 22:52:01.497087 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:01.496271 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:52:01.497087 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:01.496958 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:52:01.498580 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:01.498556 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:52:02.670403 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:02.670316 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p"] Oct 29 22:52:02.675625 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:02.675601 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" Oct 29 22:52:02.711506 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:02.710925 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pr5wj\"" Oct 29 22:52:02.711506 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:02.710969 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-99dfb0-serving-cert\"" Oct 29 22:52:02.711506 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:02.711041 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 29 22:52:02.714724 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:02.714683 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p"] Oct 29 22:52:02.743236 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:02.743126 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00fa2987-1fdc-4e03-a657-02118cb3b3e5-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p\" (UID: \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" Oct 29 22:52:02.743236 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:02.743220 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00fa2987-1fdc-4e03-a657-02118cb3b3e5-metrics-certs\") pod \"maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p\" (UID: \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" Oct 29 22:52:02.743457 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:02.743308 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmdcg\" (UniqueName: \"kubernetes.io/projected/00fa2987-1fdc-4e03-a657-02118cb3b3e5-kube-api-access-dmdcg\") pod \"maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p\" (UID: \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" Oct 29 22:52:02.743457 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:02.743333 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00fa2987-1fdc-4e03-a657-02118cb3b3e5-maas-api-tls\") pod \"maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p\" (UID: \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" Oct 29 22:52:02.844733 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:02.844696 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmdcg\" (UniqueName: \"kubernetes.io/projected/00fa2987-1fdc-4e03-a657-02118cb3b3e5-kube-api-access-dmdcg\") pod \"maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p\" (UID: \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" Oct 29 22:52:02.844916 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:02.844750 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00fa2987-1fdc-4e03-a657-02118cb3b3e5-maas-api-tls\") pod \"maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p\" (UID: \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" Oct 29 22:52:02.844916 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:02.844862 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00fa2987-1fdc-4e03-a657-02118cb3b3e5-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p\" (UID: \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" Oct 29 22:52:02.845040 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:02.844914 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00fa2987-1fdc-4e03-a657-02118cb3b3e5-metrics-certs\") pod \"maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p\" (UID: \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" Oct 29 22:52:02.845669 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:52:02.845343 5057 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-serving-cert: secret "maas-api-e2e-worker-w1-99dfb0-serving-cert" not found Oct 29 22:52:02.845669 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:52:02.845424 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00fa2987-1fdc-4e03-a657-02118cb3b3e5-maas-api-tls podName:00fa2987-1fdc-4e03-a657-02118cb3b3e5 nodeName:}" failed. No retries permitted until 2026-10-29 22:52:03.345401756 +0000 UTC m=+569.996900271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00fa2987-1fdc-4e03-a657-02118cb3b3e5-maas-api-tls") pod "maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" (UID: "00fa2987-1fdc-4e03-a657-02118cb3b3e5") : secret "maas-api-e2e-worker-w1-99dfb0-serving-cert" not found Oct 29 22:52:02.846413 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:02.846386 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00fa2987-1fdc-4e03-a657-02118cb3b3e5-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p\" (UID: \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" Oct 29 22:52:02.848291 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:02.848241 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00fa2987-1fdc-4e03-a657-02118cb3b3e5-metrics-certs\") pod \"maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p\" (UID: \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" Oct 29 22:52:02.882030 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:02.881247 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmdcg\" (UniqueName: \"kubernetes.io/projected/00fa2987-1fdc-4e03-a657-02118cb3b3e5-kube-api-access-dmdcg\") pod \"maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p\" (UID: \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" Oct 29 22:52:03.360400 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:03.360356 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00fa2987-1fdc-4e03-a657-02118cb3b3e5-maas-api-tls\") pod \"maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p\" (UID: \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" Oct 29 22:52:03.364682 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:03.364652 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00fa2987-1fdc-4e03-a657-02118cb3b3e5-maas-api-tls\") pod \"maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p\" (UID: \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" Oct 29 22:52:03.619539 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:03.619447 5057 generic.go:356] "Generic (PLEG): container finished" podID="e48474a0-2e27-4cbc-854b-382285cd2745" containerID="666b319657c565508cb82205553e9fe0df27e073ad288c820251fe527f7bbcac" exitCode=0 Oct 29 22:52:03.619716 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:03.619636 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" event={"ID":"e48474a0-2e27-4cbc-854b-382285cd2745","Type":"ContainerDied","Data":"666b319657c565508cb82205553e9fe0df27e073ad288c820251fe527f7bbcac"} Oct 29 22:52:03.620141 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:03.620117 5057 scope.go:122] "RemoveContainer" containerID="666b319657c565508cb82205553e9fe0df27e073ad288c820251fe527f7bbcac" Oct 29 22:52:03.621897 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:03.621858 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:52:03.622014 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:03.621952 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:52:03.622014 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:03.622002 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:52:03.671695 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:03.671228 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" Oct 29 22:52:04.020751 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:52:04.020688 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:52:04.059680 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:04.059393 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p"] Oct 29 22:52:04.069469 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:52:04.069376 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00fa2987_1fdc_4e03_a657_02118cb3b3e5.slice/crio-307d42eaed5806d198ef877c28a4e85117ede6275e6ebb95f791b29d4ff74e41 WatchSource:0}: Error finding container 307d42eaed5806d198ef877c28a4e85117ede6275e6ebb95f791b29d4ff74e41: Status 404 returned error can't find the container with id 307d42eaed5806d198ef877c28a4e85117ede6275e6ebb95f791b29d4ff74e41 Oct 29 22:52:04.414748 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:04.414721 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh" Oct 29 22:52:04.664255 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:04.664019 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" event={"ID":"e48474a0-2e27-4cbc-854b-382285cd2745","Type":"ContainerStarted","Data":"1971fc18e6ece09a14a14c74a8cc9bed365dc22a429f6406eeea21a9d57da17a"} Oct 29 22:52:04.667696 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:04.667667 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" event={"ID":"00fa2987-1fdc-4e03-a657-02118cb3b3e5","Type":"ContainerStarted","Data":"307d42eaed5806d198ef877c28a4e85117ede6275e6ebb95f791b29d4ff74e41"} Oct 29 22:52:05.245212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:05.244630 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:52:05.246235 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:05.246141 5057 patch_prober.go:29] interesting pod/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 29 22:52:05.246235 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:05.246211 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" podUID="e48474a0-2e27-4cbc-854b-382285cd2745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 29 22:52:05.677904 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:05.677864 5057 generic.go:356] "Generic (PLEG): container finished" podID="5fb57d7e-62d6-473b-b381-179752f820b8" containerID="6d3d48385d9d0295d8d7135770340473e4352c039924e77b1e4acd6ec9eabd75" exitCode=0 Oct 29 22:52:05.678093 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:05.678037 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6" event={"ID":"5fb57d7e-62d6-473b-b381-179752f820b8","Type":"ContainerDied","Data":"6d3d48385d9d0295d8d7135770340473e4352c039924e77b1e4acd6ec9eabd75"} Oct 29 22:52:05.682125 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:05.682090 5057 generic.go:356] "Generic (PLEG): container finished" podID="983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5" containerID="2766edb18d9b887bef2b9eff2bc2f2f130e88e08d1726221e2a3b23eeae10ed4" exitCode=0 Oct 29 22:52:05.682413 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:05.682386 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8" event={"ID":"983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5","Type":"ContainerDied","Data":"2766edb18d9b887bef2b9eff2bc2f2f130e88e08d1726221e2a3b23eeae10ed4"} Oct 29 22:52:06.232797 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:06.232667 5057 patch_prober.go:29] interesting pod/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 29 22:52:06.232797 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:06.232732 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" podUID="e48474a0-2e27-4cbc-854b-382285cd2745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 29 22:52:06.689710 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:06.689671 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" event={"ID":"00fa2987-1fdc-4e03-a657-02118cb3b3e5","Type":"ContainerStarted","Data":"77a39bdfd2ff610f1898fe1ae013015c4bcf340f97c4c366bf3edf233f9f50e7"} Oct 29 22:52:06.690174 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:06.689756 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" Oct 29 22:52:06.744220 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:06.744090 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" podStartSLOduration=2.663519371 podStartE2EDuration="4.744071526s" totalImagesPullingTime="2.080552155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:06.742769877 +0000 UTC m=+573.394268409" watchObservedRunningTime="2026-10-29 22:52:06.744071526 +0000 UTC m=+573.395570073" Oct 29 22:52:06.998219 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:06.998166 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6" Oct 29 22:52:07.043310 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.036539 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fb57d7e-62d6-473b-b381-179752f820b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fb57d7e-62d6-473b-b381-179752f820b8-maas-api-openshift-ca\") pod \"5fb57d7e-62d6-473b-b381-179752f820b8\" (UID: \"5fb57d7e-62d6-473b-b381-179752f820b8\") " Oct 29 22:52:07.043310 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.036660 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fb57d7e-62d6-473b-b381-179752f820b8-kube-api-access-lwz26\" (UniqueName: \"kubernetes.io/projected/5fb57d7e-62d6-473b-b381-179752f820b8-kube-api-access-lwz26\") pod \"5fb57d7e-62d6-473b-b381-179752f820b8\" (UID: \"5fb57d7e-62d6-473b-b381-179752f820b8\") " Oct 29 22:52:07.043310 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.036782 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fb57d7e-62d6-473b-b381-179752f820b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fb57d7e-62d6-473b-b381-179752f820b8-maas-api-xks-ca\") pod \"5fb57d7e-62d6-473b-b381-179752f820b8\" (UID: \"5fb57d7e-62d6-473b-b381-179752f820b8\") " Oct 29 22:52:07.066676 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.064589 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fb57d7e-62d6-473b-b381-179752f820b8-maas-api-openshift-ca" pod "5fb57d7e-62d6-473b-b381-179752f820b8" (UID: "5fb57d7e-62d6-473b-b381-179752f820b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:07.067346 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.067297 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fb57d7e-62d6-473b-b381-179752f820b8-maas-api-xks-ca" pod "5fb57d7e-62d6-473b-b381-179752f820b8" (UID: "5fb57d7e-62d6-473b-b381-179752f820b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:07.069994 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.069965 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb57d7e-62d6-473b-b381-179752f820b8-kube-api-access-lwz26" pod "5fb57d7e-62d6-473b-b381-179752f820b8" (UID: "5fb57d7e-62d6-473b-b381-179752f820b8"). InnerVolumeSpecName "kube-api-access-lwz26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:07.114000 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.113975 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8" Oct 29 22:52:07.145210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.143538 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-maas-api-openshift-ca\") pod \"983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5\" (UID: \"983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5\") " Oct 29 22:52:07.145210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.143592 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-kube-api-access-btj8l\" (UniqueName: \"kubernetes.io/projected/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-kube-api-access-btj8l\") pod \"983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5\" (UID: \"983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5\") " Oct 29 22:52:07.145210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.143703 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-maas-api-xks-ca\") pod \"983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5\" (UID: \"983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5\") " Oct 29 22:52:07.145210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.144063 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fb57d7e-62d6-473b-b381-179752f820b8-maas-api-openshift-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:07.145210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.144081 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwz26\" (UniqueName: \"kubernetes.io/projected/5fb57d7e-62d6-473b-b381-179752f820b8-kube-api-access-lwz26\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:07.145210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.144096 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fb57d7e-62d6-473b-b381-179752f820b8-maas-api-xks-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:07.145210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.145025 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-maas-api-openshift-ca" pod "983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5" (UID: "983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:07.153441 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.153389 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-kube-api-access-btj8l" pod "983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5" (UID: "983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5"). InnerVolumeSpecName "kube-api-access-btj8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:07.154672 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.154643 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-maas-api-xks-ca" pod "983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5" (UID: "983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:07.227265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.226675 5057 patch_prober.go:29] interesting pod/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 29 22:52:07.227265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.226738 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" podUID="e48474a0-2e27-4cbc-854b-382285cd2745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 29 22:52:07.249210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.248547 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-maas-api-xks-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:07.249210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.248585 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-maas-api-openshift-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:07.249210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.248603 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btj8l\" (UniqueName: \"kubernetes.io/projected/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5-kube-api-access-btj8l\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:07.696903 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.696870 5057 generic.go:356] "Generic (PLEG): container finished" podID="315a7746-c6a4-4565-82e0-c49ab6f02a69" containerID="ac7288d722d07173cdd3a76c508ba7d15308e9e7fdf37123bd6d9d7a2fee3c9e" exitCode=0 Oct 29 22:52:07.697398 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.696939 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" event={"ID":"315a7746-c6a4-4565-82e0-c49ab6f02a69","Type":"ContainerDied","Data":"ac7288d722d07173cdd3a76c508ba7d15308e9e7fdf37123bd6d9d7a2fee3c9e"} Oct 29 22:52:07.697398 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.697386 5057 scope.go:122] "RemoveContainer" containerID="ac7288d722d07173cdd3a76c508ba7d15308e9e7fdf37123bd6d9d7a2fee3c9e" Oct 29 22:52:07.698504 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.698489 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6" Oct 29 22:52:07.698595 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.698565 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6" event={"ID":"5fb57d7e-62d6-473b-b381-179752f820b8","Type":"ContainerDied","Data":"ca57113897700f46964faea7d3b8a4d50f113e9447f04db617072c716eb616dc"} Oct 29 22:52:07.698645 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.698613 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca57113897700f46964faea7d3b8a4d50f113e9447f04db617072c716eb616dc" Oct 29 22:52:07.699005 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.698970 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:52:07.699229 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.699051 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:52:07.699229 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.699095 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:52:07.699984 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.699959 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8" event={"ID":"983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5","Type":"ContainerDied","Data":"ba38cb0cb6ff3201ef517697a5b916f2398fdd1425747468cde3e004fb9456dd"} Oct 29 22:52:07.700080 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.699987 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8" Oct 29 22:52:07.700080 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:07.700005 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba38cb0cb6ff3201ef517697a5b916f2398fdd1425747468cde3e004fb9456dd" Oct 29 22:52:08.114902 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:08.114815 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:52:08.175207 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:08.170819 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:52:08.175511 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:08.175488 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:52:08.247212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:08.246808 5057 patch_prober.go:29] interesting pod/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 29 22:52:08.247212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:08.246898 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" podUID="e48474a0-2e27-4cbc-854b-382285cd2745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 29 22:52:08.708291 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:08.707990 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" event={"ID":"315a7746-c6a4-4565-82e0-c49ab6f02a69","Type":"ContainerStarted","Data":"a3359553fec24e3e6e002baf83b1e8c4df4741cdc0eca1f18a1e6fabb52baaf2"} Oct 29 22:52:08.840213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:08.837939 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:52:08.840213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:08.838075 5057 patch_prober.go:29] interesting pod/e2e-ait-8042816e-openshift-default-6986659497-b5fk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 29 22:52:08.840213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:08.838156 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" podUID="315a7746-c6a4-4565-82e0-c49ab6f02a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 29 22:52:09.237309 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:09.235634 5057 patch_prober.go:29] interesting pod/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 29 22:52:09.237309 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:09.235701 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" podUID="e48474a0-2e27-4cbc-854b-382285cd2745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 29 22:52:09.841348 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:09.841311 5057 patch_prober.go:29] interesting pod/e2e-ait-8042816e-openshift-default-6986659497-b5fk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 29 22:52:09.841983 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:09.841950 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" podUID="315a7746-c6a4-4565-82e0-c49ab6f02a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 29 22:52:10.106596 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:10.106488 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk"] Oct 29 22:52:10.150212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:10.149366 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw"] Oct 29 22:52:10.283060 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:10.283032 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:52:10.288256 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:10.288230 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:52:10.847209 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:10.840568 5057 patch_prober.go:29] interesting pod/e2e-ait-8042816e-openshift-default-6986659497-b5fk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 29 22:52:10.847209 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:10.840628 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" podUID="315a7746-c6a4-4565-82e0-c49ab6f02a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 29 22:52:11.842317 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:11.842282 5057 patch_prober.go:29] interesting pod/e2e-ait-8042816e-openshift-default-6986659497-b5fk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 29 22:52:11.842654 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:11.842592 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" podUID="315a7746-c6a4-4565-82e0-c49ab6f02a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 29 22:52:12.705691 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:12.705653 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" Oct 29 22:52:12.846334 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:12.846293 5057 patch_prober.go:29] interesting pod/e2e-ait-8042816e-openshift-default-6986659497-b5fk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 29 22:52:12.846973 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:12.846934 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" podUID="315a7746-c6a4-4565-82e0-c49ab6f02a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 29 22:52:13.577031 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:13.576979 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6"] Oct 29 22:52:13.592316 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:13.592283 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2bef9ca4-8wbx6"] Oct 29 22:52:13.759212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:13.757674 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m"] Oct 29 22:52:13.759212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:13.757999 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" podUID="e48474a0-2e27-4cbc-854b-382285cd2745" containerName="istio-proxy" containerID="cri-o://1971fc18e6ece09a14a14c74a8cc9bed365dc22a429f6406eeea21a9d57da17a" gracePeriod=30 Oct 29 22:52:13.843259 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:13.840267 5057 patch_prober.go:29] interesting pod/e2e-ait-8042816e-openshift-default-6986659497-b5fk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 29 22:52:13.843259 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:13.840328 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" podUID="315a7746-c6a4-4565-82e0-c49ab6f02a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 29 22:52:14.243170 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:14.243127 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm"] Oct 29 22:52:14.864854 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:14.862735 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:52:14.864854 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:14.864289 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:52:14.866449 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:14.866421 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:52:15.108243 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.108181 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk" podUID="7dc73919-716d-4f42-91c4-33c9e4f30f07" containerName="payload-processing" containerID="cri-o://34ba12aac64e08dca7241b73c43da8f0fed59318e44f077022d481eacf3e5f33" gracePeriod=25 Oct 29 22:52:15.153152 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.152298 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw" podUID="51b95244-83c6-4396-aaaf-6399a5e817b3" containerName="payload-pre-processing" containerID="cri-o://5d58c0ad1585ad92060a55c498af716ba3f93d5595fdb7c077c02fe10581a7d0" gracePeriod=25 Oct 29 22:52:15.464412 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.464360 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk" Oct 29 22:52:15.480805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.480769 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fb57d7e-62d6-473b-b381-179752f820b8" path="/var/lib/kubelet/pods/5fb57d7e-62d6-473b-b381-179752f820b8/volumes" Oct 29 22:52:15.504939 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.502866 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc73919-716d-4f42-91c4-33c9e4f30f07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc73919-716d-4f42-91c4-33c9e4f30f07-plugins-config-volume\") pod \"7dc73919-716d-4f42-91c4-33c9e4f30f07\" (UID: \"7dc73919-716d-4f42-91c4-33c9e4f30f07\") " Oct 29 22:52:15.504939 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.502946 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc73919-716d-4f42-91c4-33c9e4f30f07-kube-api-access-dd6s8\" (UniqueName: \"kubernetes.io/projected/7dc73919-716d-4f42-91c4-33c9e4f30f07-kube-api-access-dd6s8\") pod \"7dc73919-716d-4f42-91c4-33c9e4f30f07\" (UID: \"7dc73919-716d-4f42-91c4-33c9e4f30f07\") " Oct 29 22:52:15.504939 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.503559 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc73919-716d-4f42-91c4-33c9e4f30f07-plugins-config-volume" pod "7dc73919-716d-4f42-91c4-33c9e4f30f07" (UID: "7dc73919-716d-4f42-91c4-33c9e4f30f07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:15.504939 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.503802 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc73919-716d-4f42-91c4-33c9e4f30f07-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:15.521182 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.520420 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc73919-716d-4f42-91c4-33c9e4f30f07-kube-api-access-dd6s8" pod "7dc73919-716d-4f42-91c4-33c9e4f30f07" (UID: "7dc73919-716d-4f42-91c4-33c9e4f30f07"). InnerVolumeSpecName "kube-api-access-dd6s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:15.552382 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.552354 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw" Oct 29 22:52:15.605270 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.605235 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b95244-83c6-4396-aaaf-6399a5e817b3-kube-api-access-qndxm\" (UniqueName: \"kubernetes.io/projected/51b95244-83c6-4396-aaaf-6399a5e817b3-kube-api-access-qndxm\") pod \"51b95244-83c6-4396-aaaf-6399a5e817b3\" (UID: \"51b95244-83c6-4396-aaaf-6399a5e817b3\") " Oct 29 22:52:15.605434 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.605295 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51b95244-83c6-4396-aaaf-6399a5e817b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b95244-83c6-4396-aaaf-6399a5e817b3-plugins-config-volume\") pod \"51b95244-83c6-4396-aaaf-6399a5e817b3\" (UID: \"51b95244-83c6-4396-aaaf-6399a5e817b3\") " Oct 29 22:52:15.605693 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.605660 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b95244-83c6-4396-aaaf-6399a5e817b3-plugins-config-volume" pod "51b95244-83c6-4396-aaaf-6399a5e817b3" (UID: "51b95244-83c6-4396-aaaf-6399a5e817b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:15.605815 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.605793 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd6s8\" (UniqueName: \"kubernetes.io/projected/7dc73919-716d-4f42-91c4-33c9e4f30f07-kube-api-access-dd6s8\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:15.605815 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.605816 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b95244-83c6-4396-aaaf-6399a5e817b3-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:15.607500 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.607475 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b95244-83c6-4396-aaaf-6399a5e817b3-kube-api-access-qndxm" pod "51b95244-83c6-4396-aaaf-6399a5e817b3" (UID: "51b95244-83c6-4396-aaaf-6399a5e817b3"). InnerVolumeSpecName "kube-api-access-qndxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:15.683610 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.683574 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p"] Oct 29 22:52:15.684078 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.684065 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5fb57d7e-62d6-473b-b381-179752f820b8" containerName="revoke-keys" Oct 29 22:52:15.684126 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.684079 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fb57d7e-62d6-473b-b381-179752f820b8" containerName="revoke-keys" Oct 29 22:52:15.684126 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.684091 5057 cpu_manager.go:414] "containerMap: removing container" podUID="983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5" containerName="revoke-keys" Oct 29 22:52:15.684126 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.684096 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5" containerName="revoke-keys" Oct 29 22:52:15.684126 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.684115 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7dc73919-716d-4f42-91c4-33c9e4f30f07" containerName="payload-processing" Oct 29 22:52:15.684126 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.684121 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc73919-716d-4f42-91c4-33c9e4f30f07" containerName="payload-processing" Oct 29 22:52:15.684296 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.684141 5057 cpu_manager.go:414] "containerMap: removing container" podUID="51b95244-83c6-4396-aaaf-6399a5e817b3" containerName="payload-pre-processing" Oct 29 22:52:15.684296 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.684146 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b95244-83c6-4396-aaaf-6399a5e817b3" containerName="payload-pre-processing" Oct 29 22:52:15.684296 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.684231 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fb57d7e-62d6-473b-b381-179752f820b8" containerName="revoke-keys" Oct 29 22:52:15.684296 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.684239 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5" containerName="revoke-keys" Oct 29 22:52:15.684296 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.684249 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc73919-716d-4f42-91c4-33c9e4f30f07" containerName="payload-processing" Oct 29 22:52:15.684296 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.684257 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b95244-83c6-4396-aaaf-6399a5e817b3" containerName="payload-pre-processing" Oct 29 22:52:15.687614 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.687598 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p" Oct 29 22:52:15.706509 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.706478 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8042816e-vqm4p\" (UID: \"9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p" Oct 29 22:52:15.706642 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.706539 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8042816e-vqm4p\" (UID: \"9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p" Oct 29 22:52:15.706707 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.706653 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w8tx\" (UniqueName: \"kubernetes.io/projected/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-kube-api-access-8w8tx\") pod \"maas-api-revoke-keys-e2e-ait-8042816e-vqm4p\" (UID: \"9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p" Oct 29 22:52:15.706760 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.706746 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qndxm\" (UniqueName: \"kubernetes.io/projected/51b95244-83c6-4396-aaaf-6399a5e817b3-kube-api-access-qndxm\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:15.707625 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.707598 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p"] Oct 29 22:52:15.766037 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.766006 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:52:15.766212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.766067 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:52:15.783689 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.783652 5057 generic.go:356] "Generic (PLEG): container finished" podID="7dc73919-716d-4f42-91c4-33c9e4f30f07" containerID="34ba12aac64e08dca7241b73c43da8f0fed59318e44f077022d481eacf3e5f33" exitCode=0 Oct 29 22:52:15.783859 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.783696 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk" Oct 29 22:52:15.783859 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.783725 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk" event={"ID":"7dc73919-716d-4f42-91c4-33c9e4f30f07","Type":"ContainerDied","Data":"34ba12aac64e08dca7241b73c43da8f0fed59318e44f077022d481eacf3e5f33"} Oct 29 22:52:15.783859 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.783772 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk" event={"ID":"7dc73919-716d-4f42-91c4-33c9e4f30f07","Type":"ContainerDied","Data":"9081c68715bba436c3785ec3ef8dcf99bb2084c36d701597071722927430b6bf"} Oct 29 22:52:15.783859 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.783795 5057 scope.go:122] "RemoveContainer" containerID="34ba12aac64e08dca7241b73c43da8f0fed59318e44f077022d481eacf3e5f33" Oct 29 22:52:15.785295 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.785271 5057 generic.go:356] "Generic (PLEG): container finished" podID="51b95244-83c6-4396-aaaf-6399a5e817b3" containerID="5d58c0ad1585ad92060a55c498af716ba3f93d5595fdb7c077c02fe10581a7d0" exitCode=0 Oct 29 22:52:15.785420 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.785314 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw" Oct 29 22:52:15.785420 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.785354 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw" event={"ID":"51b95244-83c6-4396-aaaf-6399a5e817b3","Type":"ContainerDied","Data":"5d58c0ad1585ad92060a55c498af716ba3f93d5595fdb7c077c02fe10581a7d0"} Oct 29 22:52:15.785420 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.785405 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw" event={"ID":"51b95244-83c6-4396-aaaf-6399a5e817b3","Type":"ContainerDied","Data":"d2297efb8e962c29c7d23f2e545fca94fc1fed0396d928fe8fb1ca3016d30e91"} Oct 29 22:52:15.803877 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.803850 5057 scope.go:122] "RemoveContainer" containerID="34ba12aac64e08dca7241b73c43da8f0fed59318e44f077022d481eacf3e5f33" Oct 29 22:52:15.804286 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:52:15.804258 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34ba12aac64e08dca7241b73c43da8f0fed59318e44f077022d481eacf3e5f33\": container with ID starting with 34ba12aac64e08dca7241b73c43da8f0fed59318e44f077022d481eacf3e5f33 not found: ID does not exist" containerID="34ba12aac64e08dca7241b73c43da8f0fed59318e44f077022d481eacf3e5f33" Oct 29 22:52:15.804375 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.804297 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34ba12aac64e08dca7241b73c43da8f0fed59318e44f077022d481eacf3e5f33"} err="failed to get container status \"34ba12aac64e08dca7241b73c43da8f0fed59318e44f077022d481eacf3e5f33\": rpc error: code = NotFound desc = could not find container \"34ba12aac64e08dca7241b73c43da8f0fed59318e44f077022d481eacf3e5f33\": container with ID starting with 34ba12aac64e08dca7241b73c43da8f0fed59318e44f077022d481eacf3e5f33 not found: ID does not exist" Oct 29 22:52:15.804375 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.804322 5057 scope.go:122] "RemoveContainer" containerID="5d58c0ad1585ad92060a55c498af716ba3f93d5595fdb7c077c02fe10581a7d0" Oct 29 22:52:15.807273 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.807249 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8042816e-vqm4p\" (UID: \"9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p" Oct 29 22:52:15.807384 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.807286 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8042816e-vqm4p\" (UID: \"9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p" Oct 29 22:52:15.807384 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.807341 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w8tx\" (UniqueName: \"kubernetes.io/projected/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-kube-api-access-8w8tx\") pod \"maas-api-revoke-keys-e2e-ait-8042816e-vqm4p\" (UID: \"9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p" Oct 29 22:52:15.808026 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.808006 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8042816e-vqm4p\" (UID: \"9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p" Oct 29 22:52:15.809864 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.809831 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8042816e-vqm4p\" (UID: \"9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p" Oct 29 22:52:15.815795 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.815773 5057 scope.go:122] "RemoveContainer" containerID="5d58c0ad1585ad92060a55c498af716ba3f93d5595fdb7c077c02fe10581a7d0" Oct 29 22:52:15.816175 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:52:15.816157 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d58c0ad1585ad92060a55c498af716ba3f93d5595fdb7c077c02fe10581a7d0\": container with ID starting with 5d58c0ad1585ad92060a55c498af716ba3f93d5595fdb7c077c02fe10581a7d0 not found: ID does not exist" containerID="5d58c0ad1585ad92060a55c498af716ba3f93d5595fdb7c077c02fe10581a7d0" Oct 29 22:52:15.816347 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.816181 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d58c0ad1585ad92060a55c498af716ba3f93d5595fdb7c077c02fe10581a7d0"} err="failed to get container status \"5d58c0ad1585ad92060a55c498af716ba3f93d5595fdb7c077c02fe10581a7d0\": rpc error: code = NotFound desc = could not find container \"5d58c0ad1585ad92060a55c498af716ba3f93d5595fdb7c077c02fe10581a7d0\": container with ID starting with 5d58c0ad1585ad92060a55c498af716ba3f93d5595fdb7c077c02fe10581a7d0 not found: ID does not exist" Oct 29 22:52:15.832866 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.832841 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w8tx\" (UniqueName: \"kubernetes.io/projected/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-kube-api-access-8w8tx\") pod \"maas-api-revoke-keys-e2e-ait-8042816e-vqm4p\" (UID: \"9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p" Oct 29 22:52:15.843492 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.843464 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw"] Oct 29 22:52:15.870500 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.870445 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2bef9ca4-78d59dfcd7-fsngw"] Oct 29 22:52:15.950362 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.950264 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk"] Oct 29 22:52:15.973121 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:15.973089 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2bef9ca4-66c89f7b9b-vcbqk"] Oct 29 22:52:16.000680 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:16.000642 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p" Oct 29 22:52:16.003486 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:16.003238 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:52:16.303999 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:16.303967 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p"] Oct 29 22:52:16.305046 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:52:16.305001 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d62f776_6f79_4dfe_b8ac_eb5b2cad4a49.slice/crio-ea81864beb336e5e9310afe492764b1988c8b1d879358f985d22edcf617c53e7 WatchSource:0}: Error finding container ea81864beb336e5e9310afe492764b1988c8b1d879358f985d22edcf617c53e7: Status 404 returned error can't find the container with id ea81864beb336e5e9310afe492764b1988c8b1d879358f985d22edcf617c53e7 Oct 29 22:52:16.793856 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:16.793818 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p" event={"ID":"9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49","Type":"ContainerStarted","Data":"cfb445469a2a77718c1fc8a4dd1d57905009396ede3a9e6c124c592fde602f45"} Oct 29 22:52:16.794040 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:16.793881 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p" event={"ID":"9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49","Type":"ContainerStarted","Data":"ea81864beb336e5e9310afe492764b1988c8b1d879358f985d22edcf617c53e7"} Oct 29 22:52:16.842642 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:16.842581 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p" podStartSLOduration=1.842564002 podStartE2EDuration="1.842564002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:16.841313654 +0000 UTC m=+583.492812196" watchObservedRunningTime="2026-10-29 22:52:16.842564002 +0000 UTC m=+583.494062536" Oct 29 22:52:17.481197 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:17.481150 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51b95244-83c6-4396-aaaf-6399a5e817b3" path="/var/lib/kubelet/pods/51b95244-83c6-4396-aaaf-6399a5e817b3/volumes" Oct 29 22:52:17.481740 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:17.481719 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dc73919-716d-4f42-91c4-33c9e4f30f07" path="/var/lib/kubelet/pods/7dc73919-716d-4f42-91c4-33c9e4f30f07/volumes" Oct 29 22:52:17.802132 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:17.802037 5057 generic.go:356] "Generic (PLEG): container finished" podID="9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49" containerID="cfb445469a2a77718c1fc8a4dd1d57905009396ede3a9e6c124c592fde602f45" exitCode=0 Oct 29 22:52:17.802305 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:17.802129 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p" event={"ID":"9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49","Type":"ContainerDied","Data":"cfb445469a2a77718c1fc8a4dd1d57905009396ede3a9e6c124c592fde602f45"} Oct 29 22:52:19.014929 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.014897 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p" Oct 29 22:52:19.066447 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.066412 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-maas-api-xks-ca\") pod \"9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49\" (UID: \"9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49\") " Oct 29 22:52:19.066590 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.066515 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-maas-api-openshift-ca\") pod \"9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49\" (UID: \"9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49\") " Oct 29 22:52:19.066590 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.066556 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-kube-api-access-8w8tx\" (UniqueName: \"kubernetes.io/projected/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-kube-api-access-8w8tx\") pod \"9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49\" (UID: \"9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49\") " Oct 29 22:52:19.067536 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.067513 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-maas-api-openshift-ca" pod "9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49" (UID: "9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:19.069367 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.069337 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-kube-api-access-8w8tx" pod "9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49" (UID: "9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49"). InnerVolumeSpecName "kube-api-access-8w8tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:19.070352 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.070325 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-maas-api-xks-ca" pod "9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49" (UID: "9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:19.087030 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.086999 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:52:19.174672 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.174630 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-workload-certs\") pod \"e48474a0-2e27-4cbc-854b-382285cd2745\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " Oct 29 22:52:19.174853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.174721 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e48474a0-2e27-4cbc-854b-382285cd2745-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e48474a0-2e27-4cbc-854b-382285cd2745-istio-podinfo\") pod \"e48474a0-2e27-4cbc-854b-382285cd2745\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " Oct 29 22:52:19.174853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.174768 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e48474a0-2e27-4cbc-854b-382285cd2745-kube-api-access-zw6tr\" (UniqueName: \"kubernetes.io/projected/e48474a0-2e27-4cbc-854b-382285cd2745-kube-api-access-zw6tr\") pod \"e48474a0-2e27-4cbc-854b-382285cd2745\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " Oct 29 22:52:19.174853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.174810 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-istio-data\") pod \"e48474a0-2e27-4cbc-854b-382285cd2745\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " Oct 29 22:52:19.174853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.174843 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-istio-envoy\") pod \"e48474a0-2e27-4cbc-854b-382285cd2745\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " Oct 29 22:52:19.175085 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.174889 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e48474a0-2e27-4cbc-854b-382285cd2745-istio-token\" (UniqueName: \"kubernetes.io/projected/e48474a0-2e27-4cbc-854b-382285cd2745-istio-token\") pod \"e48474a0-2e27-4cbc-854b-382285cd2745\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " Oct 29 22:52:19.175085 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.174934 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-credential-socket\") pod \"e48474a0-2e27-4cbc-854b-382285cd2745\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " Oct 29 22:52:19.175085 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.174993 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-workload-socket\") pod \"e48474a0-2e27-4cbc-854b-382285cd2745\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " Oct 29 22:52:19.175300 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.175084 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e48474a0-2e27-4cbc-854b-382285cd2745-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e48474a0-2e27-4cbc-854b-382285cd2745-istiod-ca-cert\") pod \"e48474a0-2e27-4cbc-854b-382285cd2745\" (UID: \"e48474a0-2e27-4cbc-854b-382285cd2745\") " Oct 29 22:52:19.178553 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.176881 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-credential-socket" pod "e48474a0-2e27-4cbc-854b-382285cd2745" (UID: "e48474a0-2e27-4cbc-854b-382285cd2745"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:19.178553 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.177175 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-workload-certs" pod "e48474a0-2e27-4cbc-854b-382285cd2745" (UID: "e48474a0-2e27-4cbc-854b-382285cd2745"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:19.178553 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.178499 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-workload-socket" pod "e48474a0-2e27-4cbc-854b-382285cd2745" (UID: "e48474a0-2e27-4cbc-854b-382285cd2745"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:19.179145 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.179108 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e48474a0-2e27-4cbc-854b-382285cd2745-istiod-ca-cert" pod "e48474a0-2e27-4cbc-854b-382285cd2745" (UID: "e48474a0-2e27-4cbc-854b-382285cd2745"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:19.179369 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.179337 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-istio-data" pod "e48474a0-2e27-4cbc-854b-382285cd2745" (UID: "e48474a0-2e27-4cbc-854b-382285cd2745"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:19.187274 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.187157 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-istio-envoy" pod "e48474a0-2e27-4cbc-854b-382285cd2745" (UID: "e48474a0-2e27-4cbc-854b-382285cd2745"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:19.187403 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.187282 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48474a0-2e27-4cbc-854b-382285cd2745-istio-token" pod "e48474a0-2e27-4cbc-854b-382285cd2745" (UID: "e48474a0-2e27-4cbc-854b-382285cd2745"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:19.187833 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.187807 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48474a0-2e27-4cbc-854b-382285cd2745-kube-api-access-zw6tr" pod "e48474a0-2e27-4cbc-854b-382285cd2745" (UID: "e48474a0-2e27-4cbc-854b-382285cd2745"). InnerVolumeSpecName "kube-api-access-zw6tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:19.187929 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.187908 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e48474a0-2e27-4cbc-854b-382285cd2745-istio-podinfo" pod "e48474a0-2e27-4cbc-854b-382285cd2745" (UID: "e48474a0-2e27-4cbc-854b-382285cd2745"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:52:19.209645 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.208226 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-maas-api-openshift-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:19.209645 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.208308 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w8tx\" (UniqueName: \"kubernetes.io/projected/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-kube-api-access-8w8tx\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:19.209645 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.208330 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:19.209645 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.208348 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e48474a0-2e27-4cbc-854b-382285cd2745-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:19.209645 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.208377 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw6tr\" (UniqueName: \"kubernetes.io/projected/e48474a0-2e27-4cbc-854b-382285cd2745-kube-api-access-zw6tr\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:19.209645 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.208398 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:19.209645 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.208420 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:19.209645 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.208440 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e48474a0-2e27-4cbc-854b-382285cd2745-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:19.209645 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.208462 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:19.209645 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.208479 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48474a0-2e27-4cbc-854b-382285cd2745-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:19.209645 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.208499 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49-maas-api-xks-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:19.209645 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.208513 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e48474a0-2e27-4cbc-854b-382285cd2745-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:19.243675 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.243603 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm" podUID="da8ef65e-3b78-4373-b94e-7c54b8b7971d" containerName="payload-pre-processing" containerID="cri-o://8168a8c9353d846cc2580b7c55badf493629b2a33ee5773e26f8b52f0e6559ee" gracePeriod=25 Oct 29 22:52:19.504218 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.504174 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm" Oct 29 22:52:19.615302 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.613543 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8ef65e-3b78-4373-b94e-7c54b8b7971d-kube-api-access-x6cxm\" (UniqueName: \"kubernetes.io/projected/da8ef65e-3b78-4373-b94e-7c54b8b7971d-kube-api-access-x6cxm\") pod \"da8ef65e-3b78-4373-b94e-7c54b8b7971d\" (UID: \"da8ef65e-3b78-4373-b94e-7c54b8b7971d\") " Oct 29 22:52:19.615302 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.613734 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da8ef65e-3b78-4373-b94e-7c54b8b7971d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8ef65e-3b78-4373-b94e-7c54b8b7971d-plugins-config-volume\") pod \"da8ef65e-3b78-4373-b94e-7c54b8b7971d\" (UID: \"da8ef65e-3b78-4373-b94e-7c54b8b7971d\") " Oct 29 22:52:19.618758 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.618622 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8ef65e-3b78-4373-b94e-7c54b8b7971d-plugins-config-volume" pod "da8ef65e-3b78-4373-b94e-7c54b8b7971d" (UID: "da8ef65e-3b78-4373-b94e-7c54b8b7971d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:19.619834 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.619801 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8ef65e-3b78-4373-b94e-7c54b8b7971d-kube-api-access-x6cxm" pod "da8ef65e-3b78-4373-b94e-7c54b8b7971d" (UID: "da8ef65e-3b78-4373-b94e-7c54b8b7971d"). InnerVolumeSpecName "kube-api-access-x6cxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:19.714740 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.714701 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6cxm\" (UniqueName: \"kubernetes.io/projected/da8ef65e-3b78-4373-b94e-7c54b8b7971d-kube-api-access-x6cxm\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:19.714740 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.714737 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8ef65e-3b78-4373-b94e-7c54b8b7971d-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:19.817002 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.816969 5057 generic.go:356] "Generic (PLEG): container finished" podID="e48474a0-2e27-4cbc-854b-382285cd2745" containerID="1971fc18e6ece09a14a14c74a8cc9bed365dc22a429f6406eeea21a9d57da17a" exitCode=0 Oct 29 22:52:19.817228 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.817059 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" Oct 29 22:52:19.817228 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.817056 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" event={"ID":"e48474a0-2e27-4cbc-854b-382285cd2745","Type":"ContainerDied","Data":"1971fc18e6ece09a14a14c74a8cc9bed365dc22a429f6406eeea21a9d57da17a"} Oct 29 22:52:19.817355 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.817240 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m" event={"ID":"e48474a0-2e27-4cbc-854b-382285cd2745","Type":"ContainerDied","Data":"c912bc94d16297e328a4103e6f6fb7b03c4547f84275fe49c448301d64acf143"} Oct 29 22:52:19.817355 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.817271 5057 scope.go:122] "RemoveContainer" containerID="1971fc18e6ece09a14a14c74a8cc9bed365dc22a429f6406eeea21a9d57da17a" Oct 29 22:52:19.818819 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.818795 5057 generic.go:356] "Generic (PLEG): container finished" podID="da8ef65e-3b78-4373-b94e-7c54b8b7971d" containerID="8168a8c9353d846cc2580b7c55badf493629b2a33ee5773e26f8b52f0e6559ee" exitCode=0 Oct 29 22:52:19.818912 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.818844 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm" Oct 29 22:52:19.818912 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.818889 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm" event={"ID":"da8ef65e-3b78-4373-b94e-7c54b8b7971d","Type":"ContainerDied","Data":"8168a8c9353d846cc2580b7c55badf493629b2a33ee5773e26f8b52f0e6559ee"} Oct 29 22:52:19.819020 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.818945 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm" event={"ID":"da8ef65e-3b78-4373-b94e-7c54b8b7971d","Type":"ContainerDied","Data":"6a7325493452672067e902797e6729870941ea79956af0eb301fd5bd639a29c5"} Oct 29 22:52:19.820600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.820582 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p" event={"ID":"9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49","Type":"ContainerDied","Data":"ea81864beb336e5e9310afe492764b1988c8b1d879358f985d22edcf617c53e7"} Oct 29 22:52:19.820716 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.820615 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea81864beb336e5e9310afe492764b1988c8b1d879358f985d22edcf617c53e7" Oct 29 22:52:19.820716 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.820669 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p" Oct 29 22:52:19.827866 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.827794 5057 scope.go:122] "RemoveContainer" containerID="666b319657c565508cb82205553e9fe0df27e073ad288c820251fe527f7bbcac" Oct 29 22:52:19.837332 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.837306 5057 scope.go:122] "RemoveContainer" containerID="1971fc18e6ece09a14a14c74a8cc9bed365dc22a429f6406eeea21a9d57da17a" Oct 29 22:52:19.837634 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:52:19.837602 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1971fc18e6ece09a14a14c74a8cc9bed365dc22a429f6406eeea21a9d57da17a\": container with ID starting with 1971fc18e6ece09a14a14c74a8cc9bed365dc22a429f6406eeea21a9d57da17a not found: ID does not exist" containerID="1971fc18e6ece09a14a14c74a8cc9bed365dc22a429f6406eeea21a9d57da17a" Oct 29 22:52:19.837751 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.837633 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1971fc18e6ece09a14a14c74a8cc9bed365dc22a429f6406eeea21a9d57da17a"} err="failed to get container status \"1971fc18e6ece09a14a14c74a8cc9bed365dc22a429f6406eeea21a9d57da17a\": rpc error: code = NotFound desc = could not find container \"1971fc18e6ece09a14a14c74a8cc9bed365dc22a429f6406eeea21a9d57da17a\": container with ID starting with 1971fc18e6ece09a14a14c74a8cc9bed365dc22a429f6406eeea21a9d57da17a not found: ID does not exist" Oct 29 22:52:19.837751 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.837650 5057 scope.go:122] "RemoveContainer" containerID="666b319657c565508cb82205553e9fe0df27e073ad288c820251fe527f7bbcac" Oct 29 22:52:19.837900 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:52:19.837883 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"666b319657c565508cb82205553e9fe0df27e073ad288c820251fe527f7bbcac\": container with ID starting with 666b319657c565508cb82205553e9fe0df27e073ad288c820251fe527f7bbcac not found: ID does not exist" containerID="666b319657c565508cb82205553e9fe0df27e073ad288c820251fe527f7bbcac" Oct 29 22:52:19.837950 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.837903 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"666b319657c565508cb82205553e9fe0df27e073ad288c820251fe527f7bbcac"} err="failed to get container status \"666b319657c565508cb82205553e9fe0df27e073ad288c820251fe527f7bbcac\": rpc error: code = NotFound desc = could not find container \"666b319657c565508cb82205553e9fe0df27e073ad288c820251fe527f7bbcac\": container with ID starting with 666b319657c565508cb82205553e9fe0df27e073ad288c820251fe527f7bbcac not found: ID does not exist" Oct 29 22:52:19.837950 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.837918 5057 scope.go:122] "RemoveContainer" containerID="8168a8c9353d846cc2580b7c55badf493629b2a33ee5773e26f8b52f0e6559ee" Oct 29 22:52:19.846604 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.846585 5057 scope.go:122] "RemoveContainer" containerID="8168a8c9353d846cc2580b7c55badf493629b2a33ee5773e26f8b52f0e6559ee" Oct 29 22:52:19.846873 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:52:19.846848 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8168a8c9353d846cc2580b7c55badf493629b2a33ee5773e26f8b52f0e6559ee\": container with ID starting with 8168a8c9353d846cc2580b7c55badf493629b2a33ee5773e26f8b52f0e6559ee not found: ID does not exist" containerID="8168a8c9353d846cc2580b7c55badf493629b2a33ee5773e26f8b52f0e6559ee" Oct 29 22:52:19.846971 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.846877 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8168a8c9353d846cc2580b7c55badf493629b2a33ee5773e26f8b52f0e6559ee"} err="failed to get container status \"8168a8c9353d846cc2580b7c55badf493629b2a33ee5773e26f8b52f0e6559ee\": rpc error: code = NotFound desc = could not find container \"8168a8c9353d846cc2580b7c55badf493629b2a33ee5773e26f8b52f0e6559ee\": container with ID starting with 8168a8c9353d846cc2580b7c55badf493629b2a33ee5773e26f8b52f0e6559ee not found: ID does not exist" Oct 29 22:52:19.875768 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.875686 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm"] Oct 29 22:52:19.877772 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.877750 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e406071d-dcb44f447-xdmvm"] Oct 29 22:52:19.916283 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.916251 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m"] Oct 29 22:52:19.923815 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:19.923785 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-2bef9ca4-gw-openshift-default-dd76f9bb9-xbp4m"] Oct 29 22:52:20.567932 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:20.567897 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz"] Oct 29 22:52:21.490443 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.490404 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da8ef65e-3b78-4373-b94e-7c54b8b7971d" path="/var/lib/kubelet/pods/da8ef65e-3b78-4373-b94e-7c54b8b7971d/volumes" Oct 29 22:52:21.491074 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.491054 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e48474a0-2e27-4cbc-854b-382285cd2745" path="/var/lib/kubelet/pods/e48474a0-2e27-4cbc-854b-382285cd2745/volumes" Oct 29 22:52:21.917167 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.917083 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8"] Oct 29 22:52:21.931233 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.931145 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5db969e0-zkqg8"] Oct 29 22:52:21.935278 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.935240 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8"] Oct 29 22:52:21.935964 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.935945 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e48474a0-2e27-4cbc-854b-382285cd2745" containerName="istio-proxy" Oct 29 22:52:21.935999 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.935968 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e48474a0-2e27-4cbc-854b-382285cd2745" containerName="istio-proxy" Oct 29 22:52:21.936047 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.936030 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49" containerName="revoke-keys" Oct 29 22:52:21.936047 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.936038 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49" containerName="revoke-keys" Oct 29 22:52:21.936106 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.936052 5057 cpu_manager.go:414] "containerMap: removing container" podUID="da8ef65e-3b78-4373-b94e-7c54b8b7971d" containerName="payload-pre-processing" Oct 29 22:52:21.936106 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.936059 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8ef65e-3b78-4373-b94e-7c54b8b7971d" containerName="payload-pre-processing" Oct 29 22:52:21.936211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.936198 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e48474a0-2e27-4cbc-854b-382285cd2745" containerName="istio-proxy" Oct 29 22:52:21.936249 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.936214 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49" containerName="revoke-keys" Oct 29 22:52:21.936249 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.936223 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8ef65e-3b78-4373-b94e-7c54b8b7971d" containerName="payload-pre-processing" Oct 29 22:52:21.936249 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.936236 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e48474a0-2e27-4cbc-854b-382285cd2745" containerName="istio-proxy" Oct 29 22:52:21.941708 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.941688 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8" Oct 29 22:52:21.959037 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.958991 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-32c192\"" Oct 29 22:52:21.959584 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.959039 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-32c192-dockercfg-wxntm\"" Oct 29 22:52:21.961679 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.961598 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8"] Oct 29 22:52:21.974601 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.974565 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4"] Oct 29 22:52:21.975387 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.975365 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e48474a0-2e27-4cbc-854b-382285cd2745" containerName="istio-proxy" Oct 29 22:52:21.975462 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.975389 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e48474a0-2e27-4cbc-854b-382285cd2745" containerName="istio-proxy" Oct 29 22:52:21.979393 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:21.979370 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4" Oct 29 22:52:22.003492 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.003463 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4"] Oct 29 22:52:22.073238 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.061135 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lsdk\" (UniqueName: \"kubernetes.io/projected/6b25834c-7d18-46f7-85ea-c536e5eb3ea0-kube-api-access-8lsdk\") pod \"payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8\" (UID: \"6b25834c-7d18-46f7-85ea-c536e5eb3ea0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8" Oct 29 22:52:22.073558 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.073523 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b25834c-7d18-46f7-85ea-c536e5eb3ea0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8\" (UID: \"6b25834c-7d18-46f7-85ea-c536e5eb3ea0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8" Oct 29 22:52:22.174750 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.174659 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lsdk\" (UniqueName: \"kubernetes.io/projected/6b25834c-7d18-46f7-85ea-c536e5eb3ea0-kube-api-access-8lsdk\") pod \"payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8\" (UID: \"6b25834c-7d18-46f7-85ea-c536e5eb3ea0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8" Oct 29 22:52:22.174750 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.174705 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b25834c-7d18-46f7-85ea-c536e5eb3ea0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8\" (UID: \"6b25834c-7d18-46f7-85ea-c536e5eb3ea0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8" Oct 29 22:52:22.174967 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.174767 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpr6t\" (UniqueName: \"kubernetes.io/projected/9655e209-6b99-40ea-9602-51eceac7efa9-kube-api-access-bpr6t\") pod \"payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4\" (UID: \"9655e209-6b99-40ea-9602-51eceac7efa9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4" Oct 29 22:52:22.174967 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.174864 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9655e209-6b99-40ea-9602-51eceac7efa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4\" (UID: \"9655e209-6b99-40ea-9602-51eceac7efa9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4" Oct 29 22:52:22.175527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.175504 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b25834c-7d18-46f7-85ea-c536e5eb3ea0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8\" (UID: \"6b25834c-7d18-46f7-85ea-c536e5eb3ea0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8" Oct 29 22:52:22.200278 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.200238 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lsdk\" (UniqueName: \"kubernetes.io/projected/6b25834c-7d18-46f7-85ea-c536e5eb3ea0-kube-api-access-8lsdk\") pod \"payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8\" (UID: \"6b25834c-7d18-46f7-85ea-c536e5eb3ea0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8" Oct 29 22:52:22.262773 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.262732 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8" Oct 29 22:52:22.276533 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.276495 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9655e209-6b99-40ea-9602-51eceac7efa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4\" (UID: \"9655e209-6b99-40ea-9602-51eceac7efa9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4" Oct 29 22:52:22.276692 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.276671 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpr6t\" (UniqueName: \"kubernetes.io/projected/9655e209-6b99-40ea-9602-51eceac7efa9-kube-api-access-bpr6t\") pod \"payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4\" (UID: \"9655e209-6b99-40ea-9602-51eceac7efa9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4" Oct 29 22:52:22.277417 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.277391 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9655e209-6b99-40ea-9602-51eceac7efa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4\" (UID: \"9655e209-6b99-40ea-9602-51eceac7efa9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4" Oct 29 22:52:22.302129 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.302089 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpr6t\" (UniqueName: \"kubernetes.io/projected/9655e209-6b99-40ea-9602-51eceac7efa9-kube-api-access-bpr6t\") pod \"payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4\" (UID: \"9655e209-6b99-40ea-9602-51eceac7efa9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4" Oct 29 22:52:22.541880 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.541841 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd"] Oct 29 22:52:22.542175 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.542144 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" podUID="e8e52a93-8d37-4724-9894-43ad3a4c88b6" containerName="istio-proxy" containerID="cri-o://94b574ba71171752787e9faf697605d728602bae34ce7dd2da1ef2efa2ad05c8" gracePeriod=30 Oct 29 22:52:22.575376 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.575343 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8"] Oct 29 22:52:22.577287 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:52:22.577256 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b25834c_7d18_46f7_85ea_c536e5eb3ea0.slice/crio-d9444a1c6577dd0e04dbad858c90b56f43c52beb5f3da4f529fe294fa4536037 WatchSource:0}: Error finding container d9444a1c6577dd0e04dbad858c90b56f43c52beb5f3da4f529fe294fa4536037: Status 404 returned error can't find the container with id d9444a1c6577dd0e04dbad858c90b56f43c52beb5f3da4f529fe294fa4536037 Oct 29 22:52:22.593839 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.593792 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4" Oct 29 22:52:22.862783 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.862662 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8" event={"ID":"6b25834c-7d18-46f7-85ea-c536e5eb3ea0","Type":"ContainerStarted","Data":"3ea3cbe3dfeab5910d6a99e2ecb14c932783eae7212599dcd0df27f1658368db"} Oct 29 22:52:22.862783 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.862753 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8" event={"ID":"6b25834c-7d18-46f7-85ea-c536e5eb3ea0","Type":"ContainerStarted","Data":"d9444a1c6577dd0e04dbad858c90b56f43c52beb5f3da4f529fe294fa4536037"} Oct 29 22:52:22.863360 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.863298 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8" Oct 29 22:52:22.907460 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.907401 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8" podStartSLOduration=1.907387406 podStartE2EDuration="1.907387406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:22.907105703 +0000 UTC m=+589.558604233" watchObservedRunningTime="2026-10-29 22:52:22.907387406 +0000 UTC m=+589.558885936" Oct 29 22:52:22.949749 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:22.949716 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4"] Oct 29 22:52:22.951154 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:52:22.951128 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9655e209_6b99_40ea_9602_51eceac7efa9.slice/crio-e9b1508603c20a6333af87849331750439490240841469442b41550e47882c71 WatchSource:0}: Error finding container e9b1508603c20a6333af87849331750439490240841469442b41550e47882c71: Status 404 returned error can't find the container with id e9b1508603c20a6333af87849331750439490240841469442b41550e47882c71 Oct 29 22:52:23.480232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:23.480171 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5" path="/var/lib/kubelet/pods/983e6ca2-bd70-42bd-9b66-78cdbb0ff8f5/volumes" Oct 29 22:52:23.829132 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:23.828825 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5"] Oct 29 22:52:23.836049 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:23.836018 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:23.860217 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:23.857315 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-2077b4-openshift-default-dockercfg-4kftx\"" Oct 29 22:52:23.872718 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:23.872022 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5"] Oct 29 22:52:23.882815 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:23.882777 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4" event={"ID":"9655e209-6b99-40ea-9602-51eceac7efa9","Type":"ContainerStarted","Data":"8b2228c1e639b8054be991efcca4d2d95c608ffc068e1ccfc47aa8664cf6b69d"} Oct 29 22:52:23.882972 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:23.882868 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4" event={"ID":"9655e209-6b99-40ea-9602-51eceac7efa9","Type":"ContainerStarted","Data":"e9b1508603c20a6333af87849331750439490240841469442b41550e47882c71"} Oct 29 22:52:23.883052 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:23.883037 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4" Oct 29 22:52:23.931710 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:23.931325 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4" podStartSLOduration=2.9313092 podStartE2EDuration="2.9313092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:23.928958653 +0000 UTC m=+590.580457184" watchObservedRunningTime="2026-10-29 22:52:23.9313092 +0000 UTC m=+590.582807729" Oct 29 22:52:24.001237 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.001035 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7feb618-94bf-4988-abca-4dc4fbd699c7-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.001237 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.001080 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdc9l\" (UniqueName: \"kubernetes.io/projected/f7feb618-94bf-4988-abca-4dc4fbd699c7-kube-api-access-sdc9l\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.001807 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.001298 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.001807 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.001363 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.001807 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.001513 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.001807 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.001567 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.001807 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.001629 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.001807 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.001709 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.001807 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.001754 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.103013 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.102932 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.103013 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.102975 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.103013 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.103013 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.103327 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.103042 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.103327 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.103124 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7feb618-94bf-4988-abca-4dc4fbd699c7-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.103327 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.103164 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdc9l\" (UniqueName: \"kubernetes.io/projected/f7feb618-94bf-4988-abca-4dc4fbd699c7-kube-api-access-sdc9l\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.103327 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.103250 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.103532 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.103416 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.103532 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.103465 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.103650 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.103629 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.103695 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.103663 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.103836 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.103808 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.103944 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.103840 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.104091 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.104069 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7feb618-94bf-4988-abca-4dc4fbd699c7-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.105367 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.105350 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.105913 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.105895 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.138072 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.135380 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.138072 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.135411 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdc9l\" (UniqueName: \"kubernetes.io/projected/f7feb618-94bf-4988-abca-4dc4fbd699c7-kube-api-access-sdc9l\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.173126 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.173091 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:24.529069 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.529038 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5"] Oct 29 22:52:24.530177 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:52:24.530143 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7feb618_94bf_4988_abca_4dc4fbd699c7.slice/crio-38565599b99ba66e7a7fc79593e8da418ee184452d9fbd791f0ca292fd1983f5 WatchSource:0}: Error finding container 38565599b99ba66e7a7fc79593e8da418ee184452d9fbd791f0ca292fd1983f5: Status 404 returned error can't find the container with id 38565599b99ba66e7a7fc79593e8da418ee184452d9fbd791f0ca292fd1983f5 Oct 29 22:52:24.532698 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.532661 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:52:24.532782 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.532730 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:52:24.532782 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.532763 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:52:24.890333 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.890235 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" event={"ID":"f7feb618-94bf-4988-abca-4dc4fbd699c7","Type":"ContainerStarted","Data":"c24e4bee009e74b6b1fc9cffd573f1b02858720006716f2c2ce40d763626c965"} Oct 29 22:52:24.890333 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.890296 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" event={"ID":"f7feb618-94bf-4988-abca-4dc4fbd699c7","Type":"ContainerStarted","Data":"38565599b99ba66e7a7fc79593e8da418ee184452d9fbd791f0ca292fd1983f5"} Oct 29 22:52:24.942344 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:24.942287 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" podStartSLOduration=1.942273375 podStartE2EDuration="1.942273375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:24.939930622 +0000 UTC m=+591.591429154" watchObservedRunningTime="2026-10-29 22:52:24.942273375 +0000 UTC m=+591.593771904" Oct 29 22:52:25.174036 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:25.173962 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:25.180034 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:25.180004 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:25.569261 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:25.569180 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz" podUID="316c85fe-2455-4fad-840f-e945ab9c57d7" containerName="payload-processing" containerID="cri-o://35ed0b89ef3e0299942b82b639cc44bab37c2cfbe46278bd8a225d0793d3df01" gracePeriod=25 Oct 29 22:52:25.884001 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:25.883970 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz" Oct 29 22:52:25.897004 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:25.896969 5057 generic.go:356] "Generic (PLEG): container finished" podID="316c85fe-2455-4fad-840f-e945ab9c57d7" containerID="35ed0b89ef3e0299942b82b639cc44bab37c2cfbe46278bd8a225d0793d3df01" exitCode=0 Oct 29 22:52:25.898329 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:25.898309 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz" Oct 29 22:52:25.898517 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:25.898495 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz" event={"ID":"316c85fe-2455-4fad-840f-e945ab9c57d7","Type":"ContainerDied","Data":"35ed0b89ef3e0299942b82b639cc44bab37c2cfbe46278bd8a225d0793d3df01"} Oct 29 22:52:25.898616 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:25.898560 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz" event={"ID":"316c85fe-2455-4fad-840f-e945ab9c57d7","Type":"ContainerDied","Data":"ff331c670a05979f3866a6985156c666a2a48bf05261947aef6748bfb344e3eb"} Oct 29 22:52:25.898616 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:25.898596 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:25.898710 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:25.898630 5057 scope.go:122] "RemoveContainer" containerID="35ed0b89ef3e0299942b82b639cc44bab37c2cfbe46278bd8a225d0793d3df01" Oct 29 22:52:25.900442 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:25.900418 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:52:25.959337 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:25.959307 5057 scope.go:122] "RemoveContainer" containerID="35ed0b89ef3e0299942b82b639cc44bab37c2cfbe46278bd8a225d0793d3df01" Oct 29 22:52:25.959723 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:52:25.959702 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ed0b89ef3e0299942b82b639cc44bab37c2cfbe46278bd8a225d0793d3df01\": container with ID starting with 35ed0b89ef3e0299942b82b639cc44bab37c2cfbe46278bd8a225d0793d3df01 not found: ID does not exist" containerID="35ed0b89ef3e0299942b82b639cc44bab37c2cfbe46278bd8a225d0793d3df01" Oct 29 22:52:25.959791 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:25.959733 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ed0b89ef3e0299942b82b639cc44bab37c2cfbe46278bd8a225d0793d3df01"} err="failed to get container status \"35ed0b89ef3e0299942b82b639cc44bab37c2cfbe46278bd8a225d0793d3df01\": rpc error: code = NotFound desc = could not find container \"35ed0b89ef3e0299942b82b639cc44bab37c2cfbe46278bd8a225d0793d3df01\": container with ID starting with 35ed0b89ef3e0299942b82b639cc44bab37c2cfbe46278bd8a225d0793d3df01 not found: ID does not exist" Oct 29 22:52:26.030322 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:26.030280 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/316c85fe-2455-4fad-840f-e945ab9c57d7-kube-api-access-qxvmb\" (UniqueName: \"kubernetes.io/projected/316c85fe-2455-4fad-840f-e945ab9c57d7-kube-api-access-qxvmb\") pod \"316c85fe-2455-4fad-840f-e945ab9c57d7\" (UID: \"316c85fe-2455-4fad-840f-e945ab9c57d7\") " Oct 29 22:52:26.031112 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:26.031090 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/316c85fe-2455-4fad-840f-e945ab9c57d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/316c85fe-2455-4fad-840f-e945ab9c57d7-plugins-config-volume\") pod \"316c85fe-2455-4fad-840f-e945ab9c57d7\" (UID: \"316c85fe-2455-4fad-840f-e945ab9c57d7\") " Oct 29 22:52:26.031595 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:26.031568 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/316c85fe-2455-4fad-840f-e945ab9c57d7-plugins-config-volume" pod "316c85fe-2455-4fad-840f-e945ab9c57d7" (UID: "316c85fe-2455-4fad-840f-e945ab9c57d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:26.032048 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:26.032029 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/316c85fe-2455-4fad-840f-e945ab9c57d7-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:26.049459 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:26.049420 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/316c85fe-2455-4fad-840f-e945ab9c57d7-kube-api-access-qxvmb" pod "316c85fe-2455-4fad-840f-e945ab9c57d7" (UID: "316c85fe-2455-4fad-840f-e945ab9c57d7"). InnerVolumeSpecName "kube-api-access-qxvmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:26.148265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:26.138168 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxvmb\" (UniqueName: \"kubernetes.io/projected/316c85fe-2455-4fad-840f-e945ab9c57d7-kube-api-access-qxvmb\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:26.250815 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:26.250783 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz"] Oct 29 22:52:26.256069 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:26.256044 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-865d946f-5f594694bb-wsszz"] Oct 29 22:52:26.392287 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:26.392254 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9"] Oct 29 22:52:27.318838 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.318801 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd"] Oct 29 22:52:27.319352 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.319336 5057 cpu_manager.go:414] "containerMap: removing container" podUID="316c85fe-2455-4fad-840f-e945ab9c57d7" containerName="payload-processing" Oct 29 22:52:27.319404 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.319352 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="316c85fe-2455-4fad-840f-e945ab9c57d7" containerName="payload-processing" Oct 29 22:52:27.319443 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.319430 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="316c85fe-2455-4fad-840f-e945ab9c57d7" containerName="payload-processing" Oct 29 22:52:27.324531 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.324509 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.347612 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.347584 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd"] Oct 29 22:52:27.352239 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.350848 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.352239 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.350895 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.352239 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.350925 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.352239 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.350957 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.352239 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.351015 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.352239 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.351058 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bm7x\" (UniqueName: \"kubernetes.io/projected/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-kube-api-access-9bm7x\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.352239 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.351093 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.352239 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.351147 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.352239 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.351213 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.357765 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.357742 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-1ee571-openshift-default-dockercfg-bmxwc\"" Oct 29 22:52:27.452679 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.452647 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.452859 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.452699 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bm7x\" (UniqueName: \"kubernetes.io/projected/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-kube-api-access-9bm7x\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.452859 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.452729 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.452859 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.452775 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.452859 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.452820 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.452859 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.452852 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.453143 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.452883 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.453143 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.452915 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.453143 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.452940 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.453304 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.453267 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.453353 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.453307 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.453793 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.453769 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.453918 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.453896 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.453999 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.453916 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.455137 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.455111 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.455598 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.455565 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.482057 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.481985 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.482272 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.482253 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bm7x\" (UniqueName: \"kubernetes.io/projected/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-kube-api-access-9bm7x\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.482742 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.482719 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="316c85fe-2455-4fad-840f-e945ab9c57d7" path="/var/lib/kubelet/pods/316c85fe-2455-4fad-840f-e945ab9c57d7/volumes" Oct 29 22:52:27.642328 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.642292 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:27.829389 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.829366 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:52:27.861615 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.861540 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-token\" (UniqueName: \"kubernetes.io/projected/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-token\") pod \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " Oct 29 22:52:27.861760 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.861617 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-workload-certs\") pod \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " Oct 29 22:52:27.861760 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.861650 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-data\") pod \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " Oct 29 22:52:27.861760 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.861702 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-credential-socket\") pod \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " Oct 29 22:52:27.861926 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.861815 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e52a93-8d37-4724-9894-43ad3a4c88b6-kube-api-access-78vpb\" (UniqueName: \"kubernetes.io/projected/e8e52a93-8d37-4724-9894-43ad3a4c88b6-kube-api-access-78vpb\") pod \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " Oct 29 22:52:27.861926 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.861851 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istiod-ca-cert\") pod \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " Oct 29 22:52:27.861926 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.861908 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-workload-socket\") pod \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " Oct 29 22:52:27.862080 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.861950 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-podinfo\") pod \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " Oct 29 22:52:27.862080 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.861942 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-workload-certs" pod "e8e52a93-8d37-4724-9894-43ad3a4c88b6" (UID: "e8e52a93-8d37-4724-9894-43ad3a4c88b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:27.862080 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.861965 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-data" pod "e8e52a93-8d37-4724-9894-43ad3a4c88b6" (UID: "e8e52a93-8d37-4724-9894-43ad3a4c88b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:27.862080 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.861993 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-envoy\") pod \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\" (UID: \"e8e52a93-8d37-4724-9894-43ad3a4c88b6\") " Oct 29 22:52:27.862080 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.862015 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-credential-socket" pod "e8e52a93-8d37-4724-9894-43ad3a4c88b6" (UID: "e8e52a93-8d37-4724-9894-43ad3a4c88b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:27.862351 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.862173 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-workload-socket" pod "e8e52a93-8d37-4724-9894-43ad3a4c88b6" (UID: "e8e52a93-8d37-4724-9894-43ad3a4c88b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:27.862849 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.862637 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:27.862849 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.862661 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:27.862849 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.862678 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:27.862849 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.862692 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:27.863005 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.862871 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istiod-ca-cert" pod "e8e52a93-8d37-4724-9894-43ad3a4c88b6" (UID: "e8e52a93-8d37-4724-9894-43ad3a4c88b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:27.864559 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.864534 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-token" pod "e8e52a93-8d37-4724-9894-43ad3a4c88b6" (UID: "e8e52a93-8d37-4724-9894-43ad3a4c88b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:27.864649 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.864583 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-envoy" pod "e8e52a93-8d37-4724-9894-43ad3a4c88b6" (UID: "e8e52a93-8d37-4724-9894-43ad3a4c88b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:27.864649 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.864631 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-podinfo" pod "e8e52a93-8d37-4724-9894-43ad3a4c88b6" (UID: "e8e52a93-8d37-4724-9894-43ad3a4c88b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:52:27.865555 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.865534 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e52a93-8d37-4724-9894-43ad3a4c88b6-kube-api-access-78vpb" pod "e8e52a93-8d37-4724-9894-43ad3a4c88b6" (UID: "e8e52a93-8d37-4724-9894-43ad3a4c88b6"). InnerVolumeSpecName "kube-api-access-78vpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:27.925107 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.925070 5057 generic.go:356] "Generic (PLEG): container finished" podID="e8e52a93-8d37-4724-9894-43ad3a4c88b6" containerID="94b574ba71171752787e9faf697605d728602bae34ce7dd2da1ef2efa2ad05c8" exitCode=0 Oct 29 22:52:27.925299 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.925146 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" Oct 29 22:52:27.925299 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.925155 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" event={"ID":"e8e52a93-8d37-4724-9894-43ad3a4c88b6","Type":"ContainerDied","Data":"94b574ba71171752787e9faf697605d728602bae34ce7dd2da1ef2efa2ad05c8"} Oct 29 22:52:27.925299 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.925263 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd" event={"ID":"e8e52a93-8d37-4724-9894-43ad3a4c88b6","Type":"ContainerDied","Data":"c6315a3e8d5c9cf4673813edfd59df26af2019247baa4ca1b449815f9d83b999"} Oct 29 22:52:27.925299 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.925287 5057 scope.go:122] "RemoveContainer" containerID="94b574ba71171752787e9faf697605d728602bae34ce7dd2da1ef2efa2ad05c8" Oct 29 22:52:27.936682 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.936663 5057 scope.go:122] "RemoveContainer" containerID="811e910b203891e490acae4927c6127299e39874258fe63ebb8e7aed50ff8798" Oct 29 22:52:27.945531 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.945515 5057 scope.go:122] "RemoveContainer" containerID="94b574ba71171752787e9faf697605d728602bae34ce7dd2da1ef2efa2ad05c8" Oct 29 22:52:27.945828 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:52:27.945788 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b574ba71171752787e9faf697605d728602bae34ce7dd2da1ef2efa2ad05c8\": container with ID starting with 94b574ba71171752787e9faf697605d728602bae34ce7dd2da1ef2efa2ad05c8 not found: ID does not exist" containerID="94b574ba71171752787e9faf697605d728602bae34ce7dd2da1ef2efa2ad05c8" Oct 29 22:52:27.945869 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.945837 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b574ba71171752787e9faf697605d728602bae34ce7dd2da1ef2efa2ad05c8"} err="failed to get container status \"94b574ba71171752787e9faf697605d728602bae34ce7dd2da1ef2efa2ad05c8\": rpc error: code = NotFound desc = could not find container \"94b574ba71171752787e9faf697605d728602bae34ce7dd2da1ef2efa2ad05c8\": container with ID starting with 94b574ba71171752787e9faf697605d728602bae34ce7dd2da1ef2efa2ad05c8 not found: ID does not exist" Oct 29 22:52:27.945869 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.945853 5057 scope.go:122] "RemoveContainer" containerID="811e910b203891e490acae4927c6127299e39874258fe63ebb8e7aed50ff8798" Oct 29 22:52:27.946142 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:52:27.946120 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"811e910b203891e490acae4927c6127299e39874258fe63ebb8e7aed50ff8798\": container with ID starting with 811e910b203891e490acae4927c6127299e39874258fe63ebb8e7aed50ff8798 not found: ID does not exist" containerID="811e910b203891e490acae4927c6127299e39874258fe63ebb8e7aed50ff8798" Oct 29 22:52:27.946276 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.946151 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"811e910b203891e490acae4927c6127299e39874258fe63ebb8e7aed50ff8798"} err="failed to get container status \"811e910b203891e490acae4927c6127299e39874258fe63ebb8e7aed50ff8798\": rpc error: code = NotFound desc = could not find container \"811e910b203891e490acae4927c6127299e39874258fe63ebb8e7aed50ff8798\": container with ID starting with 811e910b203891e490acae4927c6127299e39874258fe63ebb8e7aed50ff8798 not found: ID does not exist" Oct 29 22:52:27.967224 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.967148 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:27.967224 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.967216 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:27.967224 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.967233 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:27.967741 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.967242 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78vpb\" (UniqueName: \"kubernetes.io/projected/e8e52a93-8d37-4724-9894-43ad3a4c88b6-kube-api-access-78vpb\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:27.967741 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.967252 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e52a93-8d37-4724-9894-43ad3a4c88b6-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:27.994705 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:27.986403 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd"] Oct 29 22:52:27.999959 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:52:27.999632 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326ecac6_3d7b_49ae_a584_c2b9f9aa644d.slice/crio-bf185106cdef51aa393dd1b6ee83b57ed2b45684271213e8c5b409c11db34940 WatchSource:0}: Error finding container bf185106cdef51aa393dd1b6ee83b57ed2b45684271213e8c5b409c11db34940: Status 404 returned error can't find the container with id bf185106cdef51aa393dd1b6ee83b57ed2b45684271213e8c5b409c11db34940 Oct 29 22:52:28.000263 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:28.000232 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd"] Oct 29 22:52:28.003026 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:28.002407 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:52:28.003026 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:28.002498 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:52:28.003026 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:28.002549 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:52:28.005010 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:28.004990 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5db969e0-openshift-default-6b447f5665-zsnfd"] Oct 29 22:52:28.936082 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:28.936049 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" event={"ID":"326ecac6-3d7b-49ae-a584-c2b9f9aa644d","Type":"ContainerStarted","Data":"d0ee5a238ac445666cc183f70c1329faeffd223a838497fb420aaa17c6b5ff31"} Oct 29 22:52:28.936608 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:28.936107 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" event={"ID":"326ecac6-3d7b-49ae-a584-c2b9f9aa644d","Type":"ContainerStarted","Data":"bf185106cdef51aa393dd1b6ee83b57ed2b45684271213e8c5b409c11db34940"} Oct 29 22:52:29.001951 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.001893 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" podStartSLOduration=2.001877787 podStartE2EDuration="2.001877787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:28.999592536 +0000 UTC m=+595.651091066" watchObservedRunningTime="2026-10-29 22:52:29.001877787 +0000 UTC m=+595.653376317" Oct 29 22:52:29.167545 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.167511 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr"] Oct 29 22:52:29.168374 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.168349 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e8e52a93-8d37-4724-9894-43ad3a4c88b6" containerName="istio-proxy" Oct 29 22:52:29.168474 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.168372 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e52a93-8d37-4724-9894-43ad3a4c88b6" containerName="istio-proxy" Oct 29 22:52:29.168474 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.168429 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e8e52a93-8d37-4724-9894-43ad3a4c88b6" containerName="istio-proxy" Oct 29 22:52:29.168474 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.168437 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e52a93-8d37-4724-9894-43ad3a4c88b6" containerName="istio-proxy" Oct 29 22:52:29.168627 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.168610 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e52a93-8d37-4724-9894-43ad3a4c88b6" containerName="istio-proxy" Oct 29 22:52:29.172854 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.172828 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr" Oct 29 22:52:29.189262 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.188753 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr"] Oct 29 22:52:29.192802 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.192774 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-2077b4\"" Oct 29 22:52:29.235074 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.235037 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2"] Oct 29 22:52:29.235967 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.235947 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e52a93-8d37-4724-9894-43ad3a4c88b6" containerName="istio-proxy" Oct 29 22:52:29.239689 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.239668 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2" Oct 29 22:52:29.259930 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.259879 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-2077b4-dockercfg-8bjql\"" Oct 29 22:52:29.365265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.365226 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7cvp\" (UniqueName: \"kubernetes.io/projected/e3a0a5b1-c514-4685-a97d-c75ea6882e82-kube-api-access-d7cvp\") pod \"payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2\" (UID: \"e3a0a5b1-c514-4685-a97d-c75ea6882e82\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2" Oct 29 22:52:29.366013 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.365974 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a0a5b1-c514-4685-a97d-c75ea6882e82-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2\" (UID: \"e3a0a5b1-c514-4685-a97d-c75ea6882e82\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2" Oct 29 22:52:29.366172 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.366072 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt9wn\" (UniqueName: \"kubernetes.io/projected/87936d4c-796b-4176-b54a-4b520e0b7ce5-kube-api-access-mt9wn\") pod \"payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr\" (UID: \"87936d4c-796b-4176-b54a-4b520e0b7ce5\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr" Oct 29 22:52:29.374339 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.369838 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2"] Oct 29 22:52:29.374339 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.371412 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87936d4c-796b-4176-b54a-4b520e0b7ce5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr\" (UID: \"87936d4c-796b-4176-b54a-4b520e0b7ce5\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr" Oct 29 22:52:29.390053 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.390021 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:52:29.472535 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.472447 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7cvp\" (UniqueName: \"kubernetes.io/projected/e3a0a5b1-c514-4685-a97d-c75ea6882e82-kube-api-access-d7cvp\") pod \"payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2\" (UID: \"e3a0a5b1-c514-4685-a97d-c75ea6882e82\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2" Oct 29 22:52:29.472535 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.472507 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a0a5b1-c514-4685-a97d-c75ea6882e82-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2\" (UID: \"e3a0a5b1-c514-4685-a97d-c75ea6882e82\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2" Oct 29 22:52:29.472761 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.472538 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt9wn\" (UniqueName: \"kubernetes.io/projected/87936d4c-796b-4176-b54a-4b520e0b7ce5-kube-api-access-mt9wn\") pod \"payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr\" (UID: \"87936d4c-796b-4176-b54a-4b520e0b7ce5\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr" Oct 29 22:52:29.472761 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.472628 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87936d4c-796b-4176-b54a-4b520e0b7ce5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr\" (UID: \"87936d4c-796b-4176-b54a-4b520e0b7ce5\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr" Oct 29 22:52:29.473494 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.473469 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87936d4c-796b-4176-b54a-4b520e0b7ce5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr\" (UID: \"87936d4c-796b-4176-b54a-4b520e0b7ce5\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr" Oct 29 22:52:29.473622 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.473547 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a0a5b1-c514-4685-a97d-c75ea6882e82-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2\" (UID: \"e3a0a5b1-c514-4685-a97d-c75ea6882e82\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2" Oct 29 22:52:29.483991 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.483954 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8e52a93-8d37-4724-9894-43ad3a4c88b6" path="/var/lib/kubelet/pods/e8e52a93-8d37-4724-9894-43ad3a4c88b6/volumes" Oct 29 22:52:29.497981 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.497958 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7cvp\" (UniqueName: \"kubernetes.io/projected/e3a0a5b1-c514-4685-a97d-c75ea6882e82-kube-api-access-d7cvp\") pod \"payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2\" (UID: \"e3a0a5b1-c514-4685-a97d-c75ea6882e82\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2" Oct 29 22:52:29.498261 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.498237 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt9wn\" (UniqueName: \"kubernetes.io/projected/87936d4c-796b-4176-b54a-4b520e0b7ce5-kube-api-access-mt9wn\") pod \"payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr\" (UID: \"87936d4c-796b-4176-b54a-4b520e0b7ce5\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr" Oct 29 22:52:29.555140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.555103 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2" Oct 29 22:52:29.658693 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.656795 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:29.672010 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.671895 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:29.792507 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.792425 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr" Oct 29 22:52:29.867745 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.867713 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2"] Oct 29 22:52:29.876949 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:52:29.876914 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a0a5b1_c514_4685_a97d_c75ea6882e82.slice/crio-6cee3260932de976e4d63b0c03088cc43961952e7e91d0a87ccd0ba43622b174 WatchSource:0}: Error finding container 6cee3260932de976e4d63b0c03088cc43961952e7e91d0a87ccd0ba43622b174: Status 404 returned error can't find the container with id 6cee3260932de976e4d63b0c03088cc43961952e7e91d0a87ccd0ba43622b174 Oct 29 22:52:29.944805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.944770 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2" event={"ID":"e3a0a5b1-c514-4685-a97d-c75ea6882e82","Type":"ContainerStarted","Data":"6cee3260932de976e4d63b0c03088cc43961952e7e91d0a87ccd0ba43622b174"} Oct 29 22:52:29.945243 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.945172 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:29.946700 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:29.946680 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:52:30.124956 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:52:30.124911 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87936d4c_796b_4176_b54a_4b520e0b7ce5.slice/crio-793a122a3fcbe5115e812b0d39451ba3c72271a7da8b23af282be0805489f178 WatchSource:0}: Error finding container 793a122a3fcbe5115e812b0d39451ba3c72271a7da8b23af282be0805489f178: Status 404 returned error can't find the container with id 793a122a3fcbe5115e812b0d39451ba3c72271a7da8b23af282be0805489f178 Oct 29 22:52:30.126449 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:30.126426 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr"] Oct 29 22:52:30.960699 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:30.960649 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2" event={"ID":"e3a0a5b1-c514-4685-a97d-c75ea6882e82","Type":"ContainerStarted","Data":"c4ce3d9796ce9730a641d772c9bd023d34ec70de5091cb8b3945920278671b4f"} Oct 29 22:52:30.961228 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:30.960944 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2" Oct 29 22:52:30.962721 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:30.962693 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr" event={"ID":"87936d4c-796b-4176-b54a-4b520e0b7ce5","Type":"ContainerStarted","Data":"63afa5a3a93d6f74b869d00e48fb75c41899dd8f639a9117fe3cff87b35472ce"} Oct 29 22:52:30.962844 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:30.962770 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr" Oct 29 22:52:30.962844 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:30.962804 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr" event={"ID":"87936d4c-796b-4176-b54a-4b520e0b7ce5","Type":"ContainerStarted","Data":"793a122a3fcbe5115e812b0d39451ba3c72271a7da8b23af282be0805489f178"} Oct 29 22:52:31.005996 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:31.005943 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2" podStartSLOduration=2.005927996 podStartE2EDuration="2.005927996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:31.004742749 +0000 UTC m=+597.656241280" watchObservedRunningTime="2026-10-29 22:52:31.005927996 +0000 UTC m=+597.657426526" Oct 29 22:52:31.050586 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:31.050517 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr" podStartSLOduration=2.05050214 podStartE2EDuration="2.05050214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:31.049593218 +0000 UTC m=+597.701091751" watchObservedRunningTime="2026-10-29 22:52:31.05050214 +0000 UTC m=+597.702000669" Oct 29 22:52:31.393301 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:31.393218 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9" podUID="4dc36c8e-d2aa-41d8-a72b-237d0d3f530c" containerName="payload-processing" containerID="cri-o://105ee2e5d84832f0c03c8c6de34bd9974a7182f1394c1538e3c741958ee95b6d" gracePeriod=25 Oct 29 22:52:31.678805 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:31.678777 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9" Oct 29 22:52:31.699581 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:31.699524 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dc36c8e-d2aa-41d8-a72b-237d0d3f530c-kube-api-access-9lk4q\" (UniqueName: \"kubernetes.io/projected/4dc36c8e-d2aa-41d8-a72b-237d0d3f530c-kube-api-access-9lk4q\") pod \"4dc36c8e-d2aa-41d8-a72b-237d0d3f530c\" (UID: \"4dc36c8e-d2aa-41d8-a72b-237d0d3f530c\") " Oct 29 22:52:31.699882 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:31.699670 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dc36c8e-d2aa-41d8-a72b-237d0d3f530c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc36c8e-d2aa-41d8-a72b-237d0d3f530c-plugins-config-volume\") pod \"4dc36c8e-d2aa-41d8-a72b-237d0d3f530c\" (UID: \"4dc36c8e-d2aa-41d8-a72b-237d0d3f530c\") " Oct 29 22:52:31.700491 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:31.700431 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc36c8e-d2aa-41d8-a72b-237d0d3f530c-plugins-config-volume" pod "4dc36c8e-d2aa-41d8-a72b-237d0d3f530c" (UID: "4dc36c8e-d2aa-41d8-a72b-237d0d3f530c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:31.702456 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:31.702429 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc36c8e-d2aa-41d8-a72b-237d0d3f530c-kube-api-access-9lk4q" pod "4dc36c8e-d2aa-41d8-a72b-237d0d3f530c" (UID: "4dc36c8e-d2aa-41d8-a72b-237d0d3f530c"). InnerVolumeSpecName "kube-api-access-9lk4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:31.801761 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:31.801730 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lk4q\" (UniqueName: \"kubernetes.io/projected/4dc36c8e-d2aa-41d8-a72b-237d0d3f530c-kube-api-access-9lk4q\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:31.801761 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:31.801762 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc36c8e-d2aa-41d8-a72b-237d0d3f530c-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:31.970118 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:31.970081 5057 generic.go:356] "Generic (PLEG): container finished" podID="4dc36c8e-d2aa-41d8-a72b-237d0d3f530c" containerID="105ee2e5d84832f0c03c8c6de34bd9974a7182f1394c1538e3c741958ee95b6d" exitCode=0 Oct 29 22:52:31.970118 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:31.970126 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9" Oct 29 22:52:31.970696 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:31.970178 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9" event={"ID":"4dc36c8e-d2aa-41d8-a72b-237d0d3f530c","Type":"ContainerDied","Data":"105ee2e5d84832f0c03c8c6de34bd9974a7182f1394c1538e3c741958ee95b6d"} Oct 29 22:52:31.970696 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:31.970241 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9" event={"ID":"4dc36c8e-d2aa-41d8-a72b-237d0d3f530c","Type":"ContainerDied","Data":"c8e6c58825cd8fd0dd7278afd16657ba53b21a9faf920048b065a02fb445afc7"} Oct 29 22:52:31.970696 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:31.970263 5057 scope.go:122] "RemoveContainer" containerID="105ee2e5d84832f0c03c8c6de34bd9974a7182f1394c1538e3c741958ee95b6d" Oct 29 22:52:31.980282 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:31.980267 5057 scope.go:122] "RemoveContainer" containerID="105ee2e5d84832f0c03c8c6de34bd9974a7182f1394c1538e3c741958ee95b6d" Oct 29 22:52:31.980575 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:52:31.980556 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"105ee2e5d84832f0c03c8c6de34bd9974a7182f1394c1538e3c741958ee95b6d\": container with ID starting with 105ee2e5d84832f0c03c8c6de34bd9974a7182f1394c1538e3c741958ee95b6d not found: ID does not exist" containerID="105ee2e5d84832f0c03c8c6de34bd9974a7182f1394c1538e3c741958ee95b6d" Oct 29 22:52:31.980630 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:31.980583 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"105ee2e5d84832f0c03c8c6de34bd9974a7182f1394c1538e3c741958ee95b6d"} err="failed to get container status \"105ee2e5d84832f0c03c8c6de34bd9974a7182f1394c1538e3c741958ee95b6d\": rpc error: code = NotFound desc = could not find container \"105ee2e5d84832f0c03c8c6de34bd9974a7182f1394c1538e3c741958ee95b6d\": container with ID starting with 105ee2e5d84832f0c03c8c6de34bd9974a7182f1394c1538e3c741958ee95b6d not found: ID does not exist" Oct 29 22:52:32.039212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:32.030962 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9"] Oct 29 22:52:32.051808 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:32.051753 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8042816e-655d5b57d6-9vkv9"] Oct 29 22:52:32.963361 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:32.963327 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p"] Oct 29 22:52:32.977716 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:32.977687 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8042816e-vqm4p"] Oct 29 22:52:33.482780 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:33.482750 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dc36c8e-d2aa-41d8-a72b-237d0d3f530c" path="/var/lib/kubelet/pods/4dc36c8e-d2aa-41d8-a72b-237d0d3f530c/volumes" Oct 29 22:52:33.483195 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:33.483170 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49" path="/var/lib/kubelet/pods/9d62f776-6f79-4dfe-b8ac-eb5b2cad4a49/volumes" Oct 29 22:52:33.574077 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:33.574047 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 22:52:33.574544 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:33.574514 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 22:52:33.779086 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:33.779009 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt"] Oct 29 22:52:33.779620 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:33.779605 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4dc36c8e-d2aa-41d8-a72b-237d0d3f530c" containerName="payload-processing" Oct 29 22:52:33.779694 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:33.779622 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dc36c8e-d2aa-41d8-a72b-237d0d3f530c" containerName="payload-processing" Oct 29 22:52:33.779733 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:33.779703 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dc36c8e-d2aa-41d8-a72b-237d0d3f530c" containerName="payload-processing" Oct 29 22:52:33.784694 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:33.784675 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt" Oct 29 22:52:33.807859 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:33.807822 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt"] Oct 29 22:52:33.819178 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:33.819152 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-1ee571\"" Oct 29 22:52:33.819331 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:33.819264 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-1ee571-dockercfg-nz2br\"" Oct 29 22:52:33.829165 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:33.829132 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grddx\" (UniqueName: \"kubernetes.io/projected/c9adfb1e-b9a5-4f30-9bba-ef91a855ff71-kube-api-access-grddx\") pod \"payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt\" (UID: \"c9adfb1e-b9a5-4f30-9bba-ef91a855ff71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt" Oct 29 22:52:33.829327 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:33.829246 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9adfb1e-b9a5-4f30-9bba-ef91a855ff71-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt\" (UID: \"c9adfb1e-b9a5-4f30-9bba-ef91a855ff71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt" Oct 29 22:52:33.885657 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:33.885623 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8" Oct 29 22:52:33.920025 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:52:33.919991 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:52:33.934064 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:33.934028 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9adfb1e-b9a5-4f30-9bba-ef91a855ff71-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt\" (UID: \"c9adfb1e-b9a5-4f30-9bba-ef91a855ff71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt" Oct 29 22:52:33.934275 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:33.934253 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grddx\" (UniqueName: \"kubernetes.io/projected/c9adfb1e-b9a5-4f30-9bba-ef91a855ff71-kube-api-access-grddx\") pod \"payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt\" (UID: \"c9adfb1e-b9a5-4f30-9bba-ef91a855ff71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt" Oct 29 22:52:33.935445 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:33.935414 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9adfb1e-b9a5-4f30-9bba-ef91a855ff71-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt\" (UID: \"c9adfb1e-b9a5-4f30-9bba-ef91a855ff71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt" Oct 29 22:52:34.066254 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:34.046963 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grddx\" (UniqueName: \"kubernetes.io/projected/c9adfb1e-b9a5-4f30-9bba-ef91a855ff71-kube-api-access-grddx\") pod \"payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt\" (UID: \"c9adfb1e-b9a5-4f30-9bba-ef91a855ff71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt" Oct 29 22:52:34.098238 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:34.098135 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt" Oct 29 22:52:34.404440 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:34.404275 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt"] Oct 29 22:52:34.407486 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:52:34.407434 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9adfb1e_b9a5_4f30_9bba_ef91a855ff71.slice/crio-8146b7f497a47acfef1903da801ba2c3a5d2cdc37a698f7694b9031da5c38a08 WatchSource:0}: Error finding container 8146b7f497a47acfef1903da801ba2c3a5d2cdc37a698f7694b9031da5c38a08: Status 404 returned error can't find the container with id 8146b7f497a47acfef1903da801ba2c3a5d2cdc37a698f7694b9031da5c38a08 Oct 29 22:52:34.558557 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:34.558512 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5"] Oct 29 22:52:34.558816 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:34.558788 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" podUID="315a7746-c6a4-4565-82e0-c49ab6f02a69" containerName="istio-proxy" containerID="cri-o://a3359553fec24e3e6e002baf83b1e8c4df4741cdc0eca1f18a1e6fabb52baaf2" gracePeriod=30 Oct 29 22:52:34.900210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:34.898285 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4" Oct 29 22:52:35.051651 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:35.051608 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt" event={"ID":"c9adfb1e-b9a5-4f30-9bba-ef91a855ff71","Type":"ContainerStarted","Data":"0c02e71e7d4dde6eeef528e81e1f654a41fe2c944ad6a7a01b0722fc824a5750"} Oct 29 22:52:35.051820 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:35.051679 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt" event={"ID":"c9adfb1e-b9a5-4f30-9bba-ef91a855ff71","Type":"ContainerStarted","Data":"8146b7f497a47acfef1903da801ba2c3a5d2cdc37a698f7694b9031da5c38a08"} Oct 29 22:52:35.051820 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:35.051766 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt" Oct 29 22:52:35.097566 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:35.097512 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt" podStartSLOduration=2.097498172 podStartE2EDuration="2.097498172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:35.096796354 +0000 UTC m=+601.748294885" watchObservedRunningTime="2026-10-29 22:52:35.097498172 +0000 UTC m=+601.748996717" Oct 29 22:52:35.802532 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:35.802494 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:52:36.643197 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:36.643145 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl"] Oct 29 22:52:36.647275 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:36.647257 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" Oct 29 22:52:36.674382 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:36.674343 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl"] Oct 29 22:52:36.678865 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:36.678836 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-2077b4-serving-cert\"" Oct 29 22:52:36.679026 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:36.679004 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg6ww\" (UniqueName: \"kubernetes.io/projected/e35e101c-582d-4121-9792-0f02cc8c528e-kube-api-access-vg6ww\") pod \"maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl\" (UID: \"e35e101c-582d-4121-9792-0f02cc8c528e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" Oct 29 22:52:36.679097 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:36.679072 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e35e101c-582d-4121-9792-0f02cc8c528e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl\" (UID: \"e35e101c-582d-4121-9792-0f02cc8c528e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" Oct 29 22:52:36.679246 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:36.679215 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e35e101c-582d-4121-9792-0f02cc8c528e-metrics-certs\") pod \"maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl\" (UID: \"e35e101c-582d-4121-9792-0f02cc8c528e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" Oct 29 22:52:36.679404 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:36.679360 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e35e101c-582d-4121-9792-0f02cc8c528e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl\" (UID: \"e35e101c-582d-4121-9792-0f02cc8c528e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" Oct 29 22:52:36.780565 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:36.780527 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e35e101c-582d-4121-9792-0f02cc8c528e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl\" (UID: \"e35e101c-582d-4121-9792-0f02cc8c528e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" Oct 29 22:52:36.780734 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:36.780586 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e35e101c-582d-4121-9792-0f02cc8c528e-metrics-certs\") pod \"maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl\" (UID: \"e35e101c-582d-4121-9792-0f02cc8c528e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" Oct 29 22:52:36.780734 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:36.780642 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e35e101c-582d-4121-9792-0f02cc8c528e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl\" (UID: \"e35e101c-582d-4121-9792-0f02cc8c528e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" Oct 29 22:52:36.780734 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:52:36.780681 5057 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-serving-cert: secret "maas-api-e2e-worker-w5-2077b4-serving-cert" not found Oct 29 22:52:36.780734 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:36.780721 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg6ww\" (UniqueName: \"kubernetes.io/projected/e35e101c-582d-4121-9792-0f02cc8c528e-kube-api-access-vg6ww\") pod \"maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl\" (UID: \"e35e101c-582d-4121-9792-0f02cc8c528e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" Oct 29 22:52:36.780952 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:52:36.780748 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e35e101c-582d-4121-9792-0f02cc8c528e-maas-api-tls podName:e35e101c-582d-4121-9792-0f02cc8c528e nodeName:}" failed. No retries permitted until 2026-10-29 22:52:37.280728828 +0000 UTC m=+603.932227355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e35e101c-582d-4121-9792-0f02cc8c528e-maas-api-tls") pod "maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" (UID: "e35e101c-582d-4121-9792-0f02cc8c528e") : secret "maas-api-e2e-worker-w5-2077b4-serving-cert" not found Oct 29 22:52:36.781408 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:36.781389 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e35e101c-582d-4121-9792-0f02cc8c528e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl\" (UID: \"e35e101c-582d-4121-9792-0f02cc8c528e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" Oct 29 22:52:36.783221 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:36.783160 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e35e101c-582d-4121-9792-0f02cc8c528e-metrics-certs\") pod \"maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl\" (UID: \"e35e101c-582d-4121-9792-0f02cc8c528e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" Oct 29 22:52:36.808118 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:36.808085 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg6ww\" (UniqueName: \"kubernetes.io/projected/e35e101c-582d-4121-9792-0f02cc8c528e-kube-api-access-vg6ww\") pod \"maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl\" (UID: \"e35e101c-582d-4121-9792-0f02cc8c528e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" Oct 29 22:52:37.287742 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:37.287705 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e35e101c-582d-4121-9792-0f02cc8c528e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl\" (UID: \"e35e101c-582d-4121-9792-0f02cc8c528e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" Oct 29 22:52:37.290215 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:37.290173 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e35e101c-582d-4121-9792-0f02cc8c528e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl\" (UID: \"e35e101c-582d-4121-9792-0f02cc8c528e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" Oct 29 22:52:37.561795 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:37.561707 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" Oct 29 22:52:37.872925 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:37.872845 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl"] Oct 29 22:52:37.877752 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:52:37.877726 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35e101c_582d_4121_9792_0f02cc8c528e.slice/crio-9044fe043a3a1c367532bbc19110341643566449bb78fd9ec6c6119f80bd1a6a WatchSource:0}: Error finding container 9044fe043a3a1c367532bbc19110341643566449bb78fd9ec6c6119f80bd1a6a: Status 404 returned error can't find the container with id 9044fe043a3a1c367532bbc19110341643566449bb78fd9ec6c6119f80bd1a6a Oct 29 22:52:38.087818 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:38.087779 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" event={"ID":"e35e101c-582d-4121-9792-0f02cc8c528e","Type":"ContainerStarted","Data":"9044fe043a3a1c367532bbc19110341643566449bb78fd9ec6c6119f80bd1a6a"} Oct 29 22:52:39.095900 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.095854 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" event={"ID":"e35e101c-582d-4121-9792-0f02cc8c528e","Type":"ContainerStarted","Data":"5e5733fcbe119f82d84769cf2b591ecb1cfcd2b0cfe94a71fc26b512758ca795"} Oct 29 22:52:39.096317 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.095967 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" Oct 29 22:52:39.832472 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.832407 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:52:39.914893 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.914835 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" podStartSLOduration=3.634781403 podStartE2EDuration="3.914818807s" totalImagesPullingTime="280.037404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:39.161568261 +0000 UTC m=+605.813066814" watchObservedRunningTime="2026-10-29 22:52:39.914818807 +0000 UTC m=+606.566317339" Oct 29 22:52:39.924895 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.924858 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/315a7746-c6a4-4565-82e0-c49ab6f02a69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/315a7746-c6a4-4565-82e0-c49ab6f02a69-istiod-ca-cert\") pod \"315a7746-c6a4-4565-82e0-c49ab6f02a69\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " Oct 29 22:52:39.925091 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.924960 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-podinfo\") pod \"315a7746-c6a4-4565-82e0-c49ab6f02a69\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " Oct 29 22:52:39.925091 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.924992 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-envoy\") pod \"315a7746-c6a4-4565-82e0-c49ab6f02a69\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " Oct 29 22:52:39.925091 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.925045 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-credential-socket\") pod \"315a7746-c6a4-4565-82e0-c49ab6f02a69\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " Oct 29 22:52:39.925091 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.925081 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/315a7746-c6a4-4565-82e0-c49ab6f02a69-kube-api-access-w9rsj\" (UniqueName: \"kubernetes.io/projected/315a7746-c6a4-4565-82e0-c49ab6f02a69-kube-api-access-w9rsj\") pod \"315a7746-c6a4-4565-82e0-c49ab6f02a69\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " Oct 29 22:52:39.925323 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.925174 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-workload-certs\") pod \"315a7746-c6a4-4565-82e0-c49ab6f02a69\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " Oct 29 22:52:39.925323 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.925222 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-workload-socket\") pod \"315a7746-c6a4-4565-82e0-c49ab6f02a69\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " Oct 29 22:52:39.925323 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.925267 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-data\") pod \"315a7746-c6a4-4565-82e0-c49ab6f02a69\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " Oct 29 22:52:39.925470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.925325 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-token\" (UniqueName: \"kubernetes.io/projected/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-token\") pod \"315a7746-c6a4-4565-82e0-c49ab6f02a69\" (UID: \"315a7746-c6a4-4565-82e0-c49ab6f02a69\") " Oct 29 22:52:39.925470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.925336 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/315a7746-c6a4-4565-82e0-c49ab6f02a69-istiod-ca-cert" pod "315a7746-c6a4-4565-82e0-c49ab6f02a69" (UID: "315a7746-c6a4-4565-82e0-c49ab6f02a69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:39.925470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.925422 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-credential-socket" pod "315a7746-c6a4-4565-82e0-c49ab6f02a69" (UID: "315a7746-c6a4-4565-82e0-c49ab6f02a69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:39.925689 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.925666 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-workload-socket" pod "315a7746-c6a4-4565-82e0-c49ab6f02a69" (UID: "315a7746-c6a4-4565-82e0-c49ab6f02a69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:39.925769 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.925687 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-data" pod "315a7746-c6a4-4565-82e0-c49ab6f02a69" (UID: "315a7746-c6a4-4565-82e0-c49ab6f02a69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:39.925769 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.925744 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-workload-certs" pod "315a7746-c6a4-4565-82e0-c49ab6f02a69" (UID: "315a7746-c6a4-4565-82e0-c49ab6f02a69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:39.925877 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.925781 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/315a7746-c6a4-4565-82e0-c49ab6f02a69-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:39.925877 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.925800 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:39.925877 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.925816 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:39.925877 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.925828 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:39.928246 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.928217 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-envoy" pod "315a7746-c6a4-4565-82e0-c49ab6f02a69" (UID: "315a7746-c6a4-4565-82e0-c49ab6f02a69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:39.928361 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.928272 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/315a7746-c6a4-4565-82e0-c49ab6f02a69-kube-api-access-w9rsj" pod "315a7746-c6a4-4565-82e0-c49ab6f02a69" (UID: "315a7746-c6a4-4565-82e0-c49ab6f02a69"). InnerVolumeSpecName "kube-api-access-w9rsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:39.928361 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.928289 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-token" pod "315a7746-c6a4-4565-82e0-c49ab6f02a69" (UID: "315a7746-c6a4-4565-82e0-c49ab6f02a69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:39.928498 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:39.928451 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-podinfo" pod "315a7746-c6a4-4565-82e0-c49ab6f02a69" (UID: "315a7746-c6a4-4565-82e0-c49ab6f02a69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:52:40.026821 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:40.026783 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:40.026821 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:40.026817 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:40.026821 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:40.026825 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:40.027059 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:40.026834 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9rsj\" (UniqueName: \"kubernetes.io/projected/315a7746-c6a4-4565-82e0-c49ab6f02a69-kube-api-access-w9rsj\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:40.027059 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:40.026842 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/315a7746-c6a4-4565-82e0-c49ab6f02a69-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:52:40.103893 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:40.103798 5057 generic.go:356] "Generic (PLEG): container finished" podID="315a7746-c6a4-4565-82e0-c49ab6f02a69" containerID="a3359553fec24e3e6e002baf83b1e8c4df4741cdc0eca1f18a1e6fabb52baaf2" exitCode=0 Oct 29 22:52:40.103893 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:40.103853 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" Oct 29 22:52:40.104439 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:40.103883 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" event={"ID":"315a7746-c6a4-4565-82e0-c49ab6f02a69","Type":"ContainerDied","Data":"a3359553fec24e3e6e002baf83b1e8c4df4741cdc0eca1f18a1e6fabb52baaf2"} Oct 29 22:52:40.104439 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:40.103948 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5" event={"ID":"315a7746-c6a4-4565-82e0-c49ab6f02a69","Type":"ContainerDied","Data":"895a8b31cc6c44f8c542510338f9eb2604cdc44f063e5a91052209e66a032cce"} Oct 29 22:52:40.104439 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:40.103980 5057 scope.go:122] "RemoveContainer" containerID="a3359553fec24e3e6e002baf83b1e8c4df4741cdc0eca1f18a1e6fabb52baaf2" Oct 29 22:52:40.117499 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:40.117478 5057 scope.go:122] "RemoveContainer" containerID="ac7288d722d07173cdd3a76c508ba7d15308e9e7fdf37123bd6d9d7a2fee3c9e" Oct 29 22:52:40.126397 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:40.126377 5057 scope.go:122] "RemoveContainer" containerID="a3359553fec24e3e6e002baf83b1e8c4df4741cdc0eca1f18a1e6fabb52baaf2" Oct 29 22:52:40.126763 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:52:40.126736 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3359553fec24e3e6e002baf83b1e8c4df4741cdc0eca1f18a1e6fabb52baaf2\": container with ID starting with a3359553fec24e3e6e002baf83b1e8c4df4741cdc0eca1f18a1e6fabb52baaf2 not found: ID does not exist" containerID="a3359553fec24e3e6e002baf83b1e8c4df4741cdc0eca1f18a1e6fabb52baaf2" Oct 29 22:52:40.126889 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:40.126775 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3359553fec24e3e6e002baf83b1e8c4df4741cdc0eca1f18a1e6fabb52baaf2"} err="failed to get container status \"a3359553fec24e3e6e002baf83b1e8c4df4741cdc0eca1f18a1e6fabb52baaf2\": rpc error: code = NotFound desc = could not find container \"a3359553fec24e3e6e002baf83b1e8c4df4741cdc0eca1f18a1e6fabb52baaf2\": container with ID starting with a3359553fec24e3e6e002baf83b1e8c4df4741cdc0eca1f18a1e6fabb52baaf2 not found: ID does not exist" Oct 29 22:52:40.126889 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:40.126796 5057 scope.go:122] "RemoveContainer" containerID="ac7288d722d07173cdd3a76c508ba7d15308e9e7fdf37123bd6d9d7a2fee3c9e" Oct 29 22:52:40.127106 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:52:40.127087 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac7288d722d07173cdd3a76c508ba7d15308e9e7fdf37123bd6d9d7a2fee3c9e\": container with ID starting with ac7288d722d07173cdd3a76c508ba7d15308e9e7fdf37123bd6d9d7a2fee3c9e not found: ID does not exist" containerID="ac7288d722d07173cdd3a76c508ba7d15308e9e7fdf37123bd6d9d7a2fee3c9e" Oct 29 22:52:40.127157 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:40.127114 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac7288d722d07173cdd3a76c508ba7d15308e9e7fdf37123bd6d9d7a2fee3c9e"} err="failed to get container status \"ac7288d722d07173cdd3a76c508ba7d15308e9e7fdf37123bd6d9d7a2fee3c9e\": rpc error: code = NotFound desc = could not find container \"ac7288d722d07173cdd3a76c508ba7d15308e9e7fdf37123bd6d9d7a2fee3c9e\": container with ID starting with ac7288d722d07173cdd3a76c508ba7d15308e9e7fdf37123bd6d9d7a2fee3c9e not found: ID does not exist" Oct 29 22:52:40.165202 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:40.165161 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5"] Oct 29 22:52:40.171763 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:40.171736 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8042816e-openshift-default-6986659497-b5fk5"] Oct 29 22:52:41.481121 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:41.481085 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="315a7746-c6a4-4565-82e0-c49ab6f02a69" path="/var/lib/kubelet/pods/315a7746-c6a4-4565-82e0-c49ab6f02a69/volumes" Oct 29 22:52:41.974325 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:41.974295 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr" Oct 29 22:52:41.974478 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:41.974372 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2" Oct 29 22:52:42.159858 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:42.159771 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:52:42.853946 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:42.853904 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt"] Oct 29 22:52:42.854731 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:42.854713 5057 cpu_manager.go:414] "containerMap: removing container" podUID="315a7746-c6a4-4565-82e0-c49ab6f02a69" containerName="istio-proxy" Oct 29 22:52:42.854833 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:42.854733 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="315a7746-c6a4-4565-82e0-c49ab6f02a69" containerName="istio-proxy" Oct 29 22:52:42.854932 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:42.854920 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="315a7746-c6a4-4565-82e0-c49ab6f02a69" containerName="istio-proxy" Oct 29 22:52:42.854966 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:42.854936 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="315a7746-c6a4-4565-82e0-c49ab6f02a69" containerName="istio-proxy" Oct 29 22:52:42.860378 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:42.860355 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" Oct 29 22:52:42.892340 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:42.892295 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt"] Oct 29 22:52:42.900507 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:42.900479 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-1ee571-serving-cert\"" Oct 29 22:52:42.975384 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:42.975334 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6k9p\" (UniqueName: \"kubernetes.io/projected/4ecbde51-118e-44b3-ae0c-491e4d03f124-kube-api-access-q6k9p\") pod \"maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt\" (UID: \"4ecbde51-118e-44b3-ae0c-491e4d03f124\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" Oct 29 22:52:42.975591 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:42.975451 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ecbde51-118e-44b3-ae0c-491e4d03f124-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt\" (UID: \"4ecbde51-118e-44b3-ae0c-491e4d03f124\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" Oct 29 22:52:42.975591 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:42.975493 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ecbde51-118e-44b3-ae0c-491e4d03f124-metrics-certs\") pod \"maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt\" (UID: \"4ecbde51-118e-44b3-ae0c-491e4d03f124\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" Oct 29 22:52:42.975591 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:42.975552 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ecbde51-118e-44b3-ae0c-491e4d03f124-maas-api-tls\") pod \"maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt\" (UID: \"4ecbde51-118e-44b3-ae0c-491e4d03f124\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" Oct 29 22:52:43.076378 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:43.076345 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6k9p\" (UniqueName: \"kubernetes.io/projected/4ecbde51-118e-44b3-ae0c-491e4d03f124-kube-api-access-q6k9p\") pod \"maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt\" (UID: \"4ecbde51-118e-44b3-ae0c-491e4d03f124\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" Oct 29 22:52:43.076527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:43.076416 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ecbde51-118e-44b3-ae0c-491e4d03f124-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt\" (UID: \"4ecbde51-118e-44b3-ae0c-491e4d03f124\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" Oct 29 22:52:43.076527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:43.076458 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ecbde51-118e-44b3-ae0c-491e4d03f124-metrics-certs\") pod \"maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt\" (UID: \"4ecbde51-118e-44b3-ae0c-491e4d03f124\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" Oct 29 22:52:43.076527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:43.076501 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ecbde51-118e-44b3-ae0c-491e4d03f124-maas-api-tls\") pod \"maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt\" (UID: \"4ecbde51-118e-44b3-ae0c-491e4d03f124\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" Oct 29 22:52:43.077079 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:43.077057 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ecbde51-118e-44b3-ae0c-491e4d03f124-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt\" (UID: \"4ecbde51-118e-44b3-ae0c-491e4d03f124\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" Oct 29 22:52:43.079079 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:43.079054 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ecbde51-118e-44b3-ae0c-491e4d03f124-metrics-certs\") pod \"maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt\" (UID: \"4ecbde51-118e-44b3-ae0c-491e4d03f124\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" Oct 29 22:52:43.079079 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:43.079039 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ecbde51-118e-44b3-ae0c-491e4d03f124-maas-api-tls\") pod \"maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt\" (UID: \"4ecbde51-118e-44b3-ae0c-491e4d03f124\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" Oct 29 22:52:43.105763 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:43.105700 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6k9p\" (UniqueName: \"kubernetes.io/projected/4ecbde51-118e-44b3-ae0c-491e4d03f124-kube-api-access-q6k9p\") pod \"maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt\" (UID: \"4ecbde51-118e-44b3-ae0c-491e4d03f124\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" Oct 29 22:52:43.174199 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:43.174151 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" Oct 29 22:52:43.549272 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:43.549244 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt"] Oct 29 22:52:43.550319 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:52:43.550288 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ecbde51_118e_44b3_ae0c_491e4d03f124.slice/crio-3dae2a198a93af85ecb1545fa838fc5a653683d2acfeef1221b67a82e50b9b03 WatchSource:0}: Error finding container 3dae2a198a93af85ecb1545fa838fc5a653683d2acfeef1221b67a82e50b9b03: Status 404 returned error can't find the container with id 3dae2a198a93af85ecb1545fa838fc5a653683d2acfeef1221b67a82e50b9b03 Oct 29 22:52:44.153264 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:44.153221 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" event={"ID":"4ecbde51-118e-44b3-ae0c-491e4d03f124","Type":"ContainerStarted","Data":"227156a9ee0a8dbdcce4e56fd3346310572d83d87e9c252e182e2efb56047a10"} Oct 29 22:52:44.153685 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:44.153440 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" Oct 29 22:52:44.153685 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:44.153486 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" event={"ID":"4ecbde51-118e-44b3-ae0c-491e4d03f124","Type":"ContainerStarted","Data":"3dae2a198a93af85ecb1545fa838fc5a653683d2acfeef1221b67a82e50b9b03"} Oct 29 22:52:44.203017 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:44.202958 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" podStartSLOduration=1.7935232669999999 podStartE2EDuration="2.202943246s" totalImagesPullingTime="409.419979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:44.201016408 +0000 UTC m=+610.852514940" watchObservedRunningTime="2026-10-29 22:52:44.202943246 +0000 UTC m=+610.854441777" Oct 29 22:52:45.109521 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:45.109494 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" Oct 29 22:52:46.064007 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:46.063970 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt" Oct 29 22:52:50.165655 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:50.165625 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" Oct 29 22:52:52.137210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:52.135059 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5"] Oct 29 22:52:52.137210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:52.135832 5057 cpu_manager.go:414] "containerMap: removing container" podUID="315a7746-c6a4-4565-82e0-c49ab6f02a69" containerName="istio-proxy" Oct 29 22:52:52.137210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:52.135851 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="315a7746-c6a4-4565-82e0-c49ab6f02a69" containerName="istio-proxy" Oct 29 22:52:52.164208 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:52.153476 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" Oct 29 22:52:52.192209 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:52.191821 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-ed8218-serving-cert\"" Oct 29 22:52:52.192974 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:52.192950 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5"] Oct 29 22:52:52.210461 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:52.210433 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772cd9de-1148-400d-9c3b-f0fb52455b8b-metrics-certs\") pod \"maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5\" (UID: \"772cd9de-1148-400d-9c3b-f0fb52455b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" Oct 29 22:52:52.210740 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:52.210719 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772cd9de-1148-400d-9c3b-f0fb52455b8b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5\" (UID: \"772cd9de-1148-400d-9c3b-f0fb52455b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" Oct 29 22:52:52.210901 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:52.210886 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772cd9de-1148-400d-9c3b-f0fb52455b8b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5\" (UID: \"772cd9de-1148-400d-9c3b-f0fb52455b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" Oct 29 22:52:52.212035 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:52.212009 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjq24\" (UniqueName: \"kubernetes.io/projected/772cd9de-1148-400d-9c3b-f0fb52455b8b-kube-api-access-hjq24\") pod \"maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5\" (UID: \"772cd9de-1148-400d-9c3b-f0fb52455b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" Oct 29 22:52:52.313236 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:52.313175 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772cd9de-1148-400d-9c3b-f0fb52455b8b-metrics-certs\") pod \"maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5\" (UID: \"772cd9de-1148-400d-9c3b-f0fb52455b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" Oct 29 22:52:52.313413 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:52.313321 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772cd9de-1148-400d-9c3b-f0fb52455b8b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5\" (UID: \"772cd9de-1148-400d-9c3b-f0fb52455b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" Oct 29 22:52:52.313413 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:52.313388 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772cd9de-1148-400d-9c3b-f0fb52455b8b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5\" (UID: \"772cd9de-1148-400d-9c3b-f0fb52455b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" Oct 29 22:52:52.313491 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:52.313421 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjq24\" (UniqueName: \"kubernetes.io/projected/772cd9de-1148-400d-9c3b-f0fb52455b8b-kube-api-access-hjq24\") pod \"maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5\" (UID: \"772cd9de-1148-400d-9c3b-f0fb52455b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" Oct 29 22:52:52.313852 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:52:52.313833 5057 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-serving-cert: secret "maas-api-e2e-worker-w4-ed8218-serving-cert" not found Oct 29 22:52:52.313916 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:52:52.313905 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772cd9de-1148-400d-9c3b-f0fb52455b8b-maas-api-tls podName:772cd9de-1148-400d-9c3b-f0fb52455b8b nodeName:}" failed. No retries permitted until 2026-10-29 22:52:52.8138843 +0000 UTC m=+619.465382828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/772cd9de-1148-400d-9c3b-f0fb52455b8b-maas-api-tls") pod "maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" (UID: "772cd9de-1148-400d-9c3b-f0fb52455b8b") : secret "maas-api-e2e-worker-w4-ed8218-serving-cert" not found Oct 29 22:52:52.314367 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:52.314340 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772cd9de-1148-400d-9c3b-f0fb52455b8b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5\" (UID: \"772cd9de-1148-400d-9c3b-f0fb52455b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" Oct 29 22:52:52.317020 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:52.316991 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772cd9de-1148-400d-9c3b-f0fb52455b8b-metrics-certs\") pod \"maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5\" (UID: \"772cd9de-1148-400d-9c3b-f0fb52455b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" Oct 29 22:52:52.382391 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:52.382362 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjq24\" (UniqueName: \"kubernetes.io/projected/772cd9de-1148-400d-9c3b-f0fb52455b8b-kube-api-access-hjq24\") pod \"maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5\" (UID: \"772cd9de-1148-400d-9c3b-f0fb52455b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" Oct 29 22:52:52.820376 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:52.820325 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772cd9de-1148-400d-9c3b-f0fb52455b8b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5\" (UID: \"772cd9de-1148-400d-9c3b-f0fb52455b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" Oct 29 22:52:52.857216 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:52.851080 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772cd9de-1148-400d-9c3b-f0fb52455b8b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5\" (UID: \"772cd9de-1148-400d-9c3b-f0fb52455b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" Oct 29 22:52:53.118311 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:53.117419 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" Oct 29 22:52:53.563430 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:52:53.563375 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772cd9de_1148_400d_9c3b_f0fb52455b8b.slice/crio-9b32ceedf71ced0dd17a66d29d9722d7399b664b9cd0dab83d73a67e93fec4b5 WatchSource:0}: Error finding container 9b32ceedf71ced0dd17a66d29d9722d7399b664b9cd0dab83d73a67e93fec4b5: Status 404 returned error can't find the container with id 9b32ceedf71ced0dd17a66d29d9722d7399b664b9cd0dab83d73a67e93fec4b5 Oct 29 22:52:53.571672 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:53.571602 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5"] Oct 29 22:52:54.264664 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:54.264627 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" event={"ID":"772cd9de-1148-400d-9c3b-f0fb52455b8b","Type":"ContainerStarted","Data":"c7c87fb5cef0149f8c1aa7cbe4076eef7708a2add63fc72542bf1dfe4b5313f1"} Oct 29 22:52:54.264874 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:54.264695 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" event={"ID":"772cd9de-1148-400d-9c3b-f0fb52455b8b","Type":"ContainerStarted","Data":"9b32ceedf71ced0dd17a66d29d9722d7399b664b9cd0dab83d73a67e93fec4b5"} Oct 29 22:52:54.264948 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:54.264903 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" Oct 29 22:52:54.341213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:54.323789 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:52:54.360208 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:54.354822 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" podStartSLOduration=2.043145805 podStartE2EDuration="2.35480052s" totalImagesPullingTime="311.654715ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:54.315952866 +0000 UTC m=+620.967451398" watchObservedRunningTime="2026-10-29 22:52:54.35480052 +0000 UTC m=+621.006299051" Oct 29 22:52:59.301321 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:59.301281 5057 generic.go:356] "Generic (PLEG): container finished" podID="326ecac6-3d7b-49ae-a584-c2b9f9aa644d" containerID="d0ee5a238ac445666cc183f70c1329faeffd223a838497fb420aaa17c6b5ff31" exitCode=0 Oct 29 22:52:59.301869 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:59.301355 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" event={"ID":"326ecac6-3d7b-49ae-a584-c2b9f9aa644d","Type":"ContainerDied","Data":"d0ee5a238ac445666cc183f70c1329faeffd223a838497fb420aaa17c6b5ff31"} Oct 29 22:52:59.301944 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:59.301877 5057 scope.go:122] "RemoveContainer" containerID="d0ee5a238ac445666cc183f70c1329faeffd223a838497fb420aaa17c6b5ff31" Oct 29 22:52:59.303425 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:59.303393 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:52:59.303537 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:59.303487 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:52:59.303591 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:59.303538 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:52:59.945896 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:52:59.945815 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:53:00.279949 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:00.279871 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" Oct 29 22:53:00.311109 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:00.311073 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" event={"ID":"326ecac6-3d7b-49ae-a584-c2b9f9aa644d","Type":"ContainerStarted","Data":"7612a42888170e20c7937b3a8a8f2d00b13643d07ae4855589ddbe6bfd2ab941"} Oct 29 22:53:00.644669 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:00.644543 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:53:00.646258 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:00.645930 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:53:00.646258 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:00.645984 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" podUID="326ecac6-3d7b-49ae-a584-c2b9f9aa644d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 29 22:53:01.647117 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:01.646617 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:53:01.647117 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:01.646723 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" podUID="326ecac6-3d7b-49ae-a584-c2b9f9aa644d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 29 22:53:02.354968 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:02.354932 5057 generic.go:356] "Generic (PLEG): container finished" podID="f7feb618-94bf-4988-abca-4dc4fbd699c7" containerID="c24e4bee009e74b6b1fc9cffd573f1b02858720006716f2c2ce40d763626c965" exitCode=0 Oct 29 22:53:02.355141 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:02.355009 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" event={"ID":"f7feb618-94bf-4988-abca-4dc4fbd699c7","Type":"ContainerDied","Data":"c24e4bee009e74b6b1fc9cffd573f1b02858720006716f2c2ce40d763626c965"} Oct 29 22:53:02.355608 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:02.355592 5057 scope.go:122] "RemoveContainer" containerID="c24e4bee009e74b6b1fc9cffd573f1b02858720006716f2c2ce40d763626c965" Oct 29 22:53:02.356859 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:02.356828 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:53:02.356966 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:02.356912 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:53:02.357030 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:02.356961 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:53:02.650269 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:02.649645 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:53:02.650269 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:02.649710 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" podUID="326ecac6-3d7b-49ae-a584-c2b9f9aa644d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 29 22:53:03.387736 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:03.387507 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" event={"ID":"f7feb618-94bf-4988-abca-4dc4fbd699c7","Type":"ContainerStarted","Data":"c08ef566f221f4b3fc622be50beba790a0e491d17dcd3cd69a864b16859ba433"} Oct 29 22:53:03.644268 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:03.643615 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:53:03.644268 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:03.643680 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" podUID="326ecac6-3d7b-49ae-a584-c2b9f9aa644d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 29 22:53:03.997767 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:53:03.997702 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:53:04.174210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:04.173568 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:53:04.174210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:04.173700 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:53:04.174210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:04.173744 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" podUID="f7feb618-94bf-4988-abca-4dc4fbd699c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 29 22:53:04.651211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:04.650643 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:53:04.651211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:04.650712 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" podUID="326ecac6-3d7b-49ae-a584-c2b9f9aa644d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 29 22:53:05.178754 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:05.178706 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:53:05.179330 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:05.178770 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" podUID="f7feb618-94bf-4988-abca-4dc4fbd699c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 29 22:53:05.646096 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:05.646004 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:53:05.646381 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:05.646347 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" podUID="326ecac6-3d7b-49ae-a584-c2b9f9aa644d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 29 22:53:06.173791 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:06.173748 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:53:06.173962 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:06.173817 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" podUID="f7feb618-94bf-4988-abca-4dc4fbd699c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 29 22:53:06.651838 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:06.651801 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:53:06.652264 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:06.652225 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:53:06.652880 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:06.652854 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 22:53:07.174070 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:07.174030 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:53:07.174264 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:07.174084 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" podUID="f7feb618-94bf-4988-abca-4dc4fbd699c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 29 22:53:08.177803 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:08.177769 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:53:08.178440 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:08.178403 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" podUID="f7feb618-94bf-4988-abca-4dc4fbd699c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 29 22:53:09.178162 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:09.178133 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:53:09.178669 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:09.178532 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:53:09.179098 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:09.179079 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 22:53:10.900874 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:10.900838 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:53:14.760048 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.760014 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl"] Oct 29 22:53:14.763735 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.763719 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.795124 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.795094 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-99dfb0-openshift-default-dockercfg-72qwh\"" Oct 29 22:53:14.797683 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.797664 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl"] Oct 29 22:53:14.865077 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.865038 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.865283 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.865142 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379d09bf-6153-4be5-be07-e5ba38818871-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.865283 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.865204 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd6p9\" (UniqueName: \"kubernetes.io/projected/379d09bf-6153-4be5-be07-e5ba38818871-kube-api-access-rd6p9\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.865283 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.865265 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/379d09bf-6153-4be5-be07-e5ba38818871-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.865456 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.865344 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.865456 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.865379 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379d09bf-6153-4be5-be07-e5ba38818871-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.865456 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.865430 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.865603 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.865521 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.865603 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.865578 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.966594 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.966554 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.966755 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.966604 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379d09bf-6153-4be5-be07-e5ba38818871-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.966755 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.966632 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd6p9\" (UniqueName: \"kubernetes.io/projected/379d09bf-6153-4be5-be07-e5ba38818871-kube-api-access-rd6p9\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.966755 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.966665 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/379d09bf-6153-4be5-be07-e5ba38818871-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.966755 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.966707 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.966755 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.966739 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379d09bf-6153-4be5-be07-e5ba38818871-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.967035 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.966782 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.967035 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.966822 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.967035 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.966851 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.967035 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.967005 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.967277 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.967135 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.967277 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.967237 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.967385 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.967318 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.967561 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.967541 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379d09bf-6153-4be5-be07-e5ba38818871-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.969241 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.969217 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.969331 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.969284 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379d09bf-6153-4be5-be07-e5ba38818871-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.994592 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.994558 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/379d09bf-6153-4be5-be07-e5ba38818871-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:14.994701 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:14.994665 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd6p9\" (UniqueName: \"kubernetes.io/projected/379d09bf-6153-4be5-be07-e5ba38818871-kube-api-access-rd6p9\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:15.077226 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:15.077100 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:15.388920 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:15.388890 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl"] Oct 29 22:53:15.389952 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:53:15.389929 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379d09bf_6153_4be5_be07_e5ba38818871.slice/crio-5462df73730834517cf0f9f8ebabc8219c8124247b027c1cf48c516388355b6f WatchSource:0}: Error finding container 5462df73730834517cf0f9f8ebabc8219c8124247b027c1cf48c516388355b6f: Status 404 returned error can't find the container with id 5462df73730834517cf0f9f8ebabc8219c8124247b027c1cf48c516388355b6f Oct 29 22:53:15.392359 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:15.392326 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:53:15.392430 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:15.392394 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:53:15.392468 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:15.392426 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:53:15.554947 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:15.554007 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" event={"ID":"379d09bf-6153-4be5-be07-e5ba38818871","Type":"ContainerStarted","Data":"e798c22aa47acbd74128534d604ac1e9702a34370c4c9e5fb74ac5eca5667442"} Oct 29 22:53:15.554947 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:15.554067 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" event={"ID":"379d09bf-6153-4be5-be07-e5ba38818871","Type":"ContainerStarted","Data":"5462df73730834517cf0f9f8ebabc8219c8124247b027c1cf48c516388355b6f"} Oct 29 22:53:15.604971 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:15.604909 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" podStartSLOduration=1.604894804 podStartE2EDuration="1.604894804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:15.602631683 +0000 UTC m=+642.254130213" watchObservedRunningTime="2026-10-29 22:53:15.604894804 +0000 UTC m=+642.256393334" Oct 29 22:53:16.077873 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:16.077838 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:16.079849 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:16.079814 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:53:16.079997 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:16.079869 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" podUID="379d09bf-6153-4be5-be07-e5ba38818871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 29 22:53:17.078070 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:17.077585 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:53:17.078070 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:17.077651 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" podUID="379d09bf-6153-4be5-be07-e5ba38818871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 29 22:53:18.078131 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:18.078093 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:53:18.078556 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:18.078159 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" podUID="379d09bf-6153-4be5-be07-e5ba38818871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 29 22:53:19.078299 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:19.078256 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:53:19.078702 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:19.078338 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" podUID="379d09bf-6153-4be5-be07-e5ba38818871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 29 22:53:20.078001 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:20.077966 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:53:20.078211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:20.078035 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" podUID="379d09bf-6153-4be5-be07-e5ba38818871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 29 22:53:21.077670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:21.077628 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:53:21.078151 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:21.077700 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" podUID="379d09bf-6153-4be5-be07-e5ba38818871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 29 22:53:22.077951 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:22.077915 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:53:22.078396 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:22.077967 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" podUID="379d09bf-6153-4be5-be07-e5ba38818871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 29 22:53:22.937638 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:22.937603 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:53:23.082076 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:23.082038 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:23.082465 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:23.082313 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:23.083033 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:23.083017 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 22:53:33.912916 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:53:33.912830 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:53:38.952996 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:38.952959 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:53:54.414232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:53:54.414202 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:54:03.914321 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:54:03.914280 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:54:18.937981 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:18.937939 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:54:33.913612 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:54:33.913575 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:54:44.247112 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.247073 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6"] Oct 29 22:54:44.251305 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.251279 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:44.270617 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.270592 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6"] Oct 29 22:54:44.282259 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.282231 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-99dfb0\"/\"default-dockercfg-nf4k7\"" Oct 29 22:54:44.282259 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.282239 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-99dfb0\"/\"e2e-conflict-a-03047570-kserve-self-signed-certs\"" Oct 29 22:54:44.282462 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.282330 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-99dfb0\"/\"kube-root-ca.crt\"" Oct 29 22:54:44.282462 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.282368 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-99dfb0\"/\"openshift-service-ca.crt\"" Oct 29 22:54:44.376106 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.376068 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-home\") pod \"e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:44.376106 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.376110 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-tmp-dir\") pod \"e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:44.376362 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.376133 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2d4v\" (UniqueName: \"kubernetes.io/projected/f5d589e1-b044-4b23-b722-50c764120c68-kube-api-access-d2d4v\") pod \"e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:44.376362 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.376313 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-dshm\") pod \"e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:44.376447 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.376381 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-model-cache\") pod \"e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:44.376447 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.376403 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5d589e1-b044-4b23-b722-50c764120c68-tls-certs\") pod \"e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:44.481213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.479948 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-home\") pod \"e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:44.481213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.480018 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-tmp-dir\") pod \"e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:44.481213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.480048 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2d4v\" (UniqueName: \"kubernetes.io/projected/f5d589e1-b044-4b23-b722-50c764120c68-kube-api-access-d2d4v\") pod \"e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:44.481213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.480080 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-dshm\") pod \"e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:44.481213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.480121 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-model-cache\") pod \"e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:44.481213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.480148 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5d589e1-b044-4b23-b722-50c764120c68-tls-certs\") pod \"e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:44.496243 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.488646 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-home\") pod \"e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:44.496243 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.488667 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-model-cache\") pod \"e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:44.496243 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.488910 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-tmp-dir\") pod \"e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:44.501330 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.501259 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5d589e1-b044-4b23-b722-50c764120c68-tls-certs\") pod \"e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:44.501748 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.501726 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-dshm\") pod \"e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:44.514280 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.514253 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2d4v\" (UniqueName: \"kubernetes.io/projected/f5d589e1-b044-4b23-b722-50c764120c68-kube-api-access-d2d4v\") pod \"e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:44.569917 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.569880 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:44.872471 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:44.872444 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6"] Oct 29 22:54:44.873411 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:54:44.873382 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5d589e1_b044_4b23_b722_50c764120c68.slice/crio-8fa6f827dedba84f4a4279fce5367ea2ef9a0a5987d92897ef09b481f0d80f24 WatchSource:0}: Error finding container 8fa6f827dedba84f4a4279fce5367ea2ef9a0a5987d92897ef09b481f0d80f24: Status 404 returned error can't find the container with id 8fa6f827dedba84f4a4279fce5367ea2ef9a0a5987d92897ef09b481f0d80f24 Oct 29 22:54:45.180471 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.180434 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" event={"ID":"f5d589e1-b044-4b23-b722-50c764120c68","Type":"ContainerStarted","Data":"8c2bdf4b157b60288ceec0e6b87f3c37aaaa8b9c744cb2eb202cbda2b7925f81"} Oct 29 22:54:45.180650 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.180478 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" event={"ID":"f5d589e1-b044-4b23-b722-50c764120c68","Type":"ContainerStarted","Data":"8fa6f827dedba84f4a4279fce5367ea2ef9a0a5987d92897ef09b481f0d80f24"} Oct 29 22:54:45.226057 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.226001 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" podStartSLOduration=1.225985642 podStartE2EDuration="1.225985642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:45.224502401 +0000 UTC m=+731.876000932" watchObservedRunningTime="2026-10-29 22:54:45.225985642 +0000 UTC m=+731.877484172" Oct 29 22:54:45.453458 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.453361 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql"] Oct 29 22:54:45.456307 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.456286 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.473870 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.473851 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-32c192-openshift-default-dockercfg-m9mqd\"" Oct 29 22:54:45.482248 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.482223 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql"] Oct 29 22:54:45.570811 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.570779 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:45.594421 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.594389 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66pbp\" (UniqueName: \"kubernetes.io/projected/5e20521b-2d52-4c16-bb50-48988a9671bb-kube-api-access-66pbp\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.594572 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.594430 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e20521b-2d52-4c16-bb50-48988a9671bb-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.594572 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.594538 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.594688 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.594627 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.594723 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.594705 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.594756 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.594725 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.594756 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.594746 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.594838 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.594766 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.594871 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.594855 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.696455 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.696424 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e20521b-2d52-4c16-bb50-48988a9671bb-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.696638 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.696478 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.696638 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.696501 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.696638 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.696549 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.696775 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.696747 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.696830 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.696804 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.696887 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.696841 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.696940 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.696913 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.696993 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.696958 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.697047 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.697016 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66pbp\" (UniqueName: \"kubernetes.io/projected/5e20521b-2d52-4c16-bb50-48988a9671bb-kube-api-access-66pbp\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.697205 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.697149 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.697309 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.697289 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.697366 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.697304 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e20521b-2d52-4c16-bb50-48988a9671bb-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.697366 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.697356 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.698986 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.698966 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.699130 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.699115 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.721954 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.721886 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.722097 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.722076 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66pbp\" (UniqueName: \"kubernetes.io/projected/5e20521b-2d52-4c16-bb50-48988a9671bb-kube-api-access-66pbp\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:45.771305 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:45.771278 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:46.086880 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:46.086855 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql"] Oct 29 22:54:46.088936 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:54:46.088902 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e20521b_2d52_4c16_bb50_48988a9671bb.slice/crio-5bd9f7f730ce1c9728356405dc48e67559f5de8e54ebc6aa95a9beccb2b95e31 WatchSource:0}: Error finding container 5bd9f7f730ce1c9728356405dc48e67559f5de8e54ebc6aa95a9beccb2b95e31: Status 404 returned error can't find the container with id 5bd9f7f730ce1c9728356405dc48e67559f5de8e54ebc6aa95a9beccb2b95e31 Oct 29 22:54:46.091527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:46.091495 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:54:46.091602 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:46.091558 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:54:46.091602 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:46.091588 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:54:46.189452 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:46.189413 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" event={"ID":"5e20521b-2d52-4c16-bb50-48988a9671bb","Type":"ContainerStarted","Data":"5b655bc629f6cd2bcadbd224f3e267996aa770b6a8018eecfe5f66fabbabf5b9"} Oct 29 22:54:46.189643 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:46.189473 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" event={"ID":"5e20521b-2d52-4c16-bb50-48988a9671bb","Type":"ContainerStarted","Data":"5bd9f7f730ce1c9728356405dc48e67559f5de8e54ebc6aa95a9beccb2b95e31"} Oct 29 22:54:46.771809 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:46.771768 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:47.556379 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:47.556314 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" podStartSLOduration=2.556298226 podStartE2EDuration="2.556298226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:46.242432856 +0000 UTC m=+732.893931386" watchObservedRunningTime="2026-10-29 22:54:47.556298226 +0000 UTC m=+734.207796755" Oct 29 22:54:47.557237 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:47.557209 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:54:47.773645 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:47.773600 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:54:47.774141 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:47.773661 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" podUID="5e20521b-2d52-4c16-bb50-48988a9671bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 29 22:54:48.771732 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:48.771696 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:54:48.771916 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:48.771749 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" podUID="5e20521b-2d52-4c16-bb50-48988a9671bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 29 22:54:49.772535 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:49.772499 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:54:49.772943 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:49.772552 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" podUID="5e20521b-2d52-4c16-bb50-48988a9671bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 29 22:54:50.772678 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:50.772638 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:54:50.773087 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:50.772696 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" podUID="5e20521b-2d52-4c16-bb50-48988a9671bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 29 22:54:51.777357 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:51.777319 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:54:51.777966 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:51.777930 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" podUID="5e20521b-2d52-4c16-bb50-48988a9671bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 29 22:54:53.013225 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:53.013011 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:53.013718 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:53.013573 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:53.015378 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:53.015354 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 22:54:54.571009 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:54.570968 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:54.583460 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:54.583427 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:54:55.267604 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:54:55.267578 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:55:03.915332 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:55:03.915289 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:55:04.281470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.281436 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw"] Oct 29 22:55:04.285719 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.285698 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.308759 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.308732 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw"] Oct 29 22:55:04.319491 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.319458 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-7aeaad-openshift-default-dockercfg-bxl84\"" Oct 29 22:55:04.349550 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.349513 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.349723 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.349586 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.349723 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.349624 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.349802 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.349745 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.349841 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.349808 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.349875 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.349854 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4193106b-ad8c-4e1c-bd90-3a244651d979-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.349908 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.349894 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.349941 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.349921 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.349974 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.349964 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gsqz\" (UniqueName: \"kubernetes.io/projected/4193106b-ad8c-4e1c-bd90-3a244651d979-kube-api-access-8gsqz\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.451235 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.451179 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.451430 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.451273 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.451430 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.451314 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4193106b-ad8c-4e1c-bd90-3a244651d979-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.451430 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.451347 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.451430 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.451371 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.451430 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.451410 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gsqz\" (UniqueName: \"kubernetes.io/projected/4193106b-ad8c-4e1c-bd90-3a244651d979-kube-api-access-8gsqz\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.451699 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.451508 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.451699 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.451557 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.451699 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.451601 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.451864 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.451757 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.451987 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.451965 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.452055 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.451987 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.452263 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.452243 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.452470 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.452446 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4193106b-ad8c-4e1c-bd90-3a244651d979-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.454076 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.454056 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.454223 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.454175 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.482553 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.482526 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.482878 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.482858 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gsqz\" (UniqueName: \"kubernetes.io/projected/4193106b-ad8c-4e1c-bd90-3a244651d979-kube-api-access-8gsqz\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.601098 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.601006 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:04.949414 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:55:04.949340 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4193106b_ad8c_4e1c_bd90_3a244651d979.slice/crio-ddf74988d336d72c634b1818d6dd077c86e71b8def59d042272fdeb105ca6de6 WatchSource:0}: Error finding container ddf74988d336d72c634b1818d6dd077c86e71b8def59d042272fdeb105ca6de6: Status 404 returned error can't find the container with id ddf74988d336d72c634b1818d6dd077c86e71b8def59d042272fdeb105ca6de6 Oct 29 22:55:04.949918 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.949881 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw"] Oct 29 22:55:04.953987 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.953944 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:55:04.954104 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.954057 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:55:04.954167 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:04.954104 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:55:05.326488 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:05.326385 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" event={"ID":"4193106b-ad8c-4e1c-bd90-3a244651d979","Type":"ContainerStarted","Data":"be3117ff97f131af4ed72af5c41127f838cd9b074db501bd2a02a93594085ab0"} Oct 29 22:55:05.326488 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:05.326456 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" event={"ID":"4193106b-ad8c-4e1c-bd90-3a244651d979","Type":"ContainerStarted","Data":"ddf74988d336d72c634b1818d6dd077c86e71b8def59d042272fdeb105ca6de6"} Oct 29 22:55:05.391578 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:05.391518 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" podStartSLOduration=1.391503745 podStartE2EDuration="1.391503745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:05.389462443 +0000 UTC m=+752.040960978" watchObservedRunningTime="2026-10-29 22:55:05.391503745 +0000 UTC m=+752.043002275" Oct 29 22:55:05.601559 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:05.601488 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:06.603733 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:06.603694 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:55:06.604144 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:06.603748 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" podUID="4193106b-ad8c-4e1c-bd90-3a244651d979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 29 22:55:07.601925 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:07.601888 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:55:07.602100 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:07.601945 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" podUID="4193106b-ad8c-4e1c-bd90-3a244651d979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 29 22:55:08.152264 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:08.151856 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:55:08.602231 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:08.602197 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:55:08.602412 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:08.602255 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" podUID="4193106b-ad8c-4e1c-bd90-3a244651d979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 29 22:55:09.602404 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:09.601950 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:55:09.602404 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:09.602014 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" podUID="4193106b-ad8c-4e1c-bd90-3a244651d979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 29 22:55:10.601678 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:10.601639 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:55:10.601848 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:10.601707 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" podUID="4193106b-ad8c-4e1c-bd90-3a244651d979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 29 22:55:11.601543 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:11.601501 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:55:11.602073 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:11.601561 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" podUID="4193106b-ad8c-4e1c-bd90-3a244651d979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 29 22:55:12.605686 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:12.605656 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:12.606181 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:12.606038 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:12.606793 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:12.606775 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 22:55:26.093301 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.093211 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd"] Oct 29 22:55:26.097633 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.097610 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:26.119379 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.119353 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd"] Oct 29 22:55:26.128854 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.128820 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-99dfb0\"/\"e2e-conflict-b-03047570-kserve-self-signed-certs\"" Oct 29 22:55:26.209118 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.209080 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8qs4\" (UniqueName: \"kubernetes.io/projected/908e8092-13d9-443e-93c3-5ba58c6626c7-kube-api-access-g8qs4\") pod \"e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:26.209281 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.209232 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908e8092-13d9-443e-93c3-5ba58c6626c7-tls-certs\") pod \"e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:26.209281 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.209262 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-model-cache\") pod \"e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:26.209373 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.209360 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-dshm\") pod \"e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:26.209423 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.209391 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-home\") pod \"e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:26.209460 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.209428 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-tmp-dir\") pod \"e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:26.314213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.313827 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8qs4\" (UniqueName: \"kubernetes.io/projected/908e8092-13d9-443e-93c3-5ba58c6626c7-kube-api-access-g8qs4\") pod \"e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:26.314213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.313951 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908e8092-13d9-443e-93c3-5ba58c6626c7-tls-certs\") pod \"e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:26.314213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.313995 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-model-cache\") pod \"e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:26.314213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.314080 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-dshm\") pod \"e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:26.314213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.314114 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-home\") pod \"e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:26.314213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.314173 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-tmp-dir\") pod \"e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:26.314730 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.314702 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-tmp-dir\") pod \"e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:26.322577 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.322545 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-dshm\") pod \"e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:26.334028 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.333959 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908e8092-13d9-443e-93c3-5ba58c6626c7-tls-certs\") pod \"e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:26.340367 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.337465 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-model-cache\") pod \"e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:26.340367 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.338420 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-home\") pod \"e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:26.352257 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.352104 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8qs4\" (UniqueName: \"kubernetes.io/projected/908e8092-13d9-443e-93c3-5ba58c6626c7-kube-api-access-g8qs4\") pod \"e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:26.424378 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.424313 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:26.741338 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:26.741286 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd"] Oct 29 22:55:26.741713 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:55:26.741670 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod908e8092_13d9_443e_93c3_5ba58c6626c7.slice/crio-af847851d9aa6c1310298cf84514a4accb613c2cdb97d580bd1833e85d65d487 WatchSource:0}: Error finding container af847851d9aa6c1310298cf84514a4accb613c2cdb97d580bd1833e85d65d487: Status 404 returned error can't find the container with id af847851d9aa6c1310298cf84514a4accb613c2cdb97d580bd1833e85d65d487 Oct 29 22:55:27.532213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:27.516482 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" event={"ID":"908e8092-13d9-443e-93c3-5ba58c6626c7","Type":"ContainerStarted","Data":"2f1512582f5a4e085ff16a8554089d2e3f3925805956d0cbbf0555e3e3d75442"} Oct 29 22:55:27.532213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:27.516559 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" event={"ID":"908e8092-13d9-443e-93c3-5ba58c6626c7","Type":"ContainerStarted","Data":"af847851d9aa6c1310298cf84514a4accb613c2cdb97d580bd1833e85d65d487"} Oct 29 22:55:27.569067 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:27.569005 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" podStartSLOduration=1.5689893270000002 podStartE2EDuration="1.568989327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:27.568749146 +0000 UTC m=+774.220247675" watchObservedRunningTime="2026-10-29 22:55:27.568989327 +0000 UTC m=+774.220487859" Oct 29 22:55:28.424731 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:28.424696 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:33.913105 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:55:33.913063 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:55:36.424638 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:36.424604 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:36.437399 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:36.437373 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:36.585050 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:36.585019 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:55:40.746210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:40.744121 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:55:49.071734 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.071697 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s"] Oct 29 22:55:49.077507 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.077487 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:49.097111 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.097085 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s"] Oct 29 22:55:49.107887 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.107860 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2077b4\"/\"openshift-service-ca.crt\"" Oct 29 22:55:49.108240 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.108215 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2077b4\"/\"kube-root-ca.crt\"" Oct 29 22:55:49.108422 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.108401 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2077b4\"/\"default-dockercfg-v8lk4\"" Oct 29 22:55:49.108537 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.108499 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2077b4\"/\"distinct-2-w5-2077b4-kserve-self-signed-certs\"" Oct 29 22:55:49.225538 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.225503 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a56baba-0899-4946-89d0-13db7c5c0242-tls-certs\") pod \"distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:49.225719 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.225561 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-dshm\") pod \"distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:49.225719 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.225694 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-model-cache\") pod \"distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:49.225792 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.225729 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkb5h\" (UniqueName: \"kubernetes.io/projected/4a56baba-0899-4946-89d0-13db7c5c0242-kube-api-access-zkb5h\") pod \"distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:49.225869 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.225847 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-tmp-dir\") pod \"distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:49.225983 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.225967 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-home\") pod \"distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:49.337271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.333848 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-home\") pod \"distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:49.337271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.333965 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a56baba-0899-4946-89d0-13db7c5c0242-tls-certs\") pod \"distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:49.337271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.334017 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-dshm\") pod \"distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:49.337271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.334074 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-model-cache\") pod \"distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:49.337271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.334096 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkb5h\" (UniqueName: \"kubernetes.io/projected/4a56baba-0899-4946-89d0-13db7c5c0242-kube-api-access-zkb5h\") pod \"distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:49.337271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.334143 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-tmp-dir\") pod \"distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:49.337271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.334669 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-tmp-dir\") pod \"distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:49.337271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.334894 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-home\") pod \"distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:49.337271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.335031 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-model-cache\") pod \"distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:49.337271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.337095 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-dshm\") pod \"distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:49.352409 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.352376 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a56baba-0899-4946-89d0-13db7c5c0242-tls-certs\") pod \"distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:49.365992 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.365963 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkb5h\" (UniqueName: \"kubernetes.io/projected/4a56baba-0899-4946-89d0-13db7c5c0242-kube-api-access-zkb5h\") pod \"distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:49.390201 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.390164 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:49.699415 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:49.699386 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s"] Oct 29 22:55:49.699541 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:55:49.699462 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a56baba_0899_4946_89d0_13db7c5c0242.slice/crio-c6c98f01045aef1a65629f7deb867b4560841222b7a312dc106ed91907ddf704 WatchSource:0}: Error finding container c6c98f01045aef1a65629f7deb867b4560841222b7a312dc106ed91907ddf704: Status 404 returned error can't find the container with id c6c98f01045aef1a65629f7deb867b4560841222b7a312dc106ed91907ddf704 Oct 29 22:55:50.710287 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:50.710235 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" event={"ID":"4a56baba-0899-4946-89d0-13db7c5c0242","Type":"ContainerStarted","Data":"c2644bec3bdd682a4dace47ed469a669ef075214e6a59213b77e3226e04865ba"} Oct 29 22:55:50.710755 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:50.710296 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" event={"ID":"4a56baba-0899-4946-89d0-13db7c5c0242","Type":"ContainerStarted","Data":"c6c98f01045aef1a65629f7deb867b4560841222b7a312dc106ed91907ddf704"} Oct 29 22:55:51.390287 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.390257 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:51.476853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.476772 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podStartSLOduration=2.476753737 podStartE2EDuration="2.476753737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:50.757320523 +0000 UTC m=+797.408819053" watchObservedRunningTime="2026-10-29 22:55:51.476753737 +0000 UTC m=+798.128252268" Oct 29 22:55:51.481853 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.481825 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq"] Oct 29 22:55:51.486551 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.486530 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.510800 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.510770 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq"] Oct 29 22:55:51.520116 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.520080 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ed8218-openshift-default-dockercfg-x28tf\"" Oct 29 22:55:51.662735 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.662640 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.662735 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.662695 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w2pm\" (UniqueName: \"kubernetes.io/projected/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-kube-api-access-7w2pm\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.662967 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.662789 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.662967 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.662818 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.662967 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.662849 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.662967 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.662934 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.663175 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.663033 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.663175 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.663073 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.663175 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.663101 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.764335 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.764296 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.764811 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.764361 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.764811 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.764391 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.764811 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.764546 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.764811 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.764585 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w2pm\" (UniqueName: \"kubernetes.io/projected/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-kube-api-access-7w2pm\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.764811 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.764621 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.764811 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.764651 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.764811 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.764691 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.764811 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.764717 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.764811 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.764796 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.765508 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.765480 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.765862 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.765840 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.766066 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.765911 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.766213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.766096 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.768466 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.768421 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.768594 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.768571 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.790869 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.790837 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w2pm\" (UniqueName: \"kubernetes.io/projected/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-kube-api-access-7w2pm\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.791158 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.791133 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:51.801069 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:51.801044 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:52.128053 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:52.128019 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq"] Oct 29 22:55:52.129758 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:55:52.129713 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5a1a45_f903_4dcb_8d39_fabd9bf226e4.slice/crio-18f20be1ac6c1986864551d2b8b4e4bcabd60e20b8a59d9817e6a6ada0d9cdae WatchSource:0}: Error finding container 18f20be1ac6c1986864551d2b8b4e4bcabd60e20b8a59d9817e6a6ada0d9cdae: Status 404 returned error can't find the container with id 18f20be1ac6c1986864551d2b8b4e4bcabd60e20b8a59d9817e6a6ada0d9cdae Oct 29 22:55:52.132568 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:52.132522 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:55:52.132656 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:52.132615 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:55:52.132710 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:52.132658 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:55:52.733212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:52.729890 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" event={"ID":"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4","Type":"ContainerStarted","Data":"bd9108463c34b48f8ec48e2128b8955fc67f20ada8e4809004bc00c5a4e5b3f8"} Oct 29 22:55:52.733212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:52.729951 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" event={"ID":"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4","Type":"ContainerStarted","Data":"18f20be1ac6c1986864551d2b8b4e4bcabd60e20b8a59d9817e6a6ada0d9cdae"} Oct 29 22:55:52.784984 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:52.784933 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" podStartSLOduration=1.7849173409999999 podStartE2EDuration="1.784917341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:52.782907605 +0000 UTC m=+799.434406136" watchObservedRunningTime="2026-10-29 22:55:52.784917341 +0000 UTC m=+799.436415939" Oct 29 22:55:52.801778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:52.801748 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:53.803484 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:53.803448 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 29 22:55:53.803950 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:53.803503 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" podUID="ea5a1a45-f903-4dcb-8d39-fabd9bf226e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 29 22:55:54.801720 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:54.801687 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 29 22:55:54.801903 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:54.801741 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" podUID="ea5a1a45-f903-4dcb-8d39-fabd9bf226e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 29 22:55:55.801885 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:55.801848 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 29 22:55:55.802315 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:55.801914 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" podUID="ea5a1a45-f903-4dcb-8d39-fabd9bf226e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 29 22:55:56.802060 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:56.802023 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 29 22:55:56.802583 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:56.802099 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" podUID="ea5a1a45-f903-4dcb-8d39-fabd9bf226e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 29 22:55:57.802377 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:57.802345 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 29 22:55:57.802779 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:57.802395 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" podUID="ea5a1a45-f903-4dcb-8d39-fabd9bf226e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 29 22:55:59.176320 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:59.176284 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:59.176743 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:59.176664 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:59.177385 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:59.177364 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 22:55:59.390420 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:59.390374 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:59.414638 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:59.414607 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:55:59.809645 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:55:59.809619 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:56:01.395141 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:01.395113 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd"] Oct 29 22:56:03.913133 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:56:03.913092 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:56:06.960018 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:06.959978 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:56:13.877023 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:13.876986 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc"] Oct 29 22:56:13.883228 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:13.883203 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:13.900769 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:13.900735 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc"] Oct 29 22:56:13.911679 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:13.911647 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-1ee571\"/\"openshift-service-ca.crt\"" Oct 29 22:56:13.917546 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:13.917516 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-1ee571\"/\"default-dockercfg-2lnwl\"" Oct 29 22:56:13.917546 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:13.917526 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-1ee571\"/\"distinct-2-w6-1ee571-kserve-self-signed-certs\"" Oct 29 22:56:13.917713 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:13.917528 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-1ee571\"/\"kube-root-ca.crt\"" Oct 29 22:56:13.973014 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:13.972979 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-home\") pod \"distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:13.973174 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:13.973023 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrr22\" (UniqueName: \"kubernetes.io/projected/017ea90c-95e4-4199-8f90-e67ee5f80c41-kube-api-access-nrr22\") pod \"distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:13.973174 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:13.973088 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017ea90c-95e4-4199-8f90-e67ee5f80c41-tls-certs\") pod \"distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:13.973174 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:13.973109 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-model-cache\") pod \"distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:13.973174 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:13.973158 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-tmp-dir\") pod \"distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:13.973347 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:13.973303 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-dshm\") pod \"distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:14.078041 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:14.077996 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-tmp-dir\") pod \"distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:14.078992 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:14.078968 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-dshm\") pod \"distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:14.079265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:14.079247 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-home\") pod \"distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:14.079405 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:14.079389 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrr22\" (UniqueName: \"kubernetes.io/projected/017ea90c-95e4-4199-8f90-e67ee5f80c41-kube-api-access-nrr22\") pod \"distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:14.090206 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:14.079532 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017ea90c-95e4-4199-8f90-e67ee5f80c41-tls-certs\") pod \"distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:14.091543 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:14.078823 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-tmp-dir\") pod \"distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:14.091543 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:14.090695 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-home\") pod \"distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:14.092099 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:14.092057 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-model-cache\") pod \"distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:14.092238 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:14.091461 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-model-cache\") pod \"distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:14.109810 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:14.108844 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-dshm\") pod \"distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:14.117249 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:14.110784 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017ea90c-95e4-4199-8f90-e67ee5f80c41-tls-certs\") pod \"distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:14.158295 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:14.157720 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrr22\" (UniqueName: \"kubernetes.io/projected/017ea90c-95e4-4199-8f90-e67ee5f80c41-kube-api-access-nrr22\") pod \"distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:14.199631 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:14.199593 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:14.504757 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:14.504728 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc"] Oct 29 22:56:14.505053 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:56:14.505031 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod017ea90c_95e4_4199_8f90_e67ee5f80c41.slice/crio-c8219443fa4caf57a2e0d754f7d2f29c2b9dea1f877422403b1d2d9345f4861a WatchSource:0}: Error finding container c8219443fa4caf57a2e0d754f7d2f29c2b9dea1f877422403b1d2d9345f4861a: Status 404 returned error can't find the container with id c8219443fa4caf57a2e0d754f7d2f29c2b9dea1f877422403b1d2d9345f4861a Oct 29 22:56:14.930271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:14.930236 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" event={"ID":"017ea90c-95e4-4199-8f90-e67ee5f80c41","Type":"ContainerStarted","Data":"e884af9ba506e29c0f0414cc733eb6e50bdcc757b1eb46ed735235d9aceb115a"} Oct 29 22:56:14.930726 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:14.930285 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" event={"ID":"017ea90c-95e4-4199-8f90-e67ee5f80c41","Type":"ContainerStarted","Data":"c8219443fa4caf57a2e0d754f7d2f29c2b9dea1f877422403b1d2d9345f4861a"} Oct 29 22:56:14.976998 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:14.976940 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podStartSLOduration=1.976926558 podStartE2EDuration="1.976926558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:14.974900925 +0000 UTC m=+821.626399451" watchObservedRunningTime="2026-10-29 22:56:14.976926558 +0000 UTC m=+821.628425087" Oct 29 22:56:15.200745 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:15.200671 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:15.290425 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:15.290362 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6"] Oct 29 22:56:16.418073 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.418034 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" podUID="908e8092-13d9-443e-93c3-5ba58c6626c7" containerName="main" containerID="cri-o://2f1512582f5a4e085ff16a8554089d2e3f3925805956d0cbbf0555e3e3d75442" gracePeriod=45 Oct 29 22:56:16.574993 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.574952 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" podUID="908e8092-13d9-443e-93c3-5ba58c6626c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": dial tcp 10.132.0.79:8000: connect: connection refused" Oct 29 22:56:16.722519 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.722492 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:56:16.837807 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.837742 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/908e8092-13d9-443e-93c3-5ba58c6626c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/908e8092-13d9-443e-93c3-5ba58c6626c7-tls-certs\") pod \"908e8092-13d9-443e-93c3-5ba58c6626c7\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " Oct 29 22:56:16.838050 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.837814 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-tmp-dir\") pod \"908e8092-13d9-443e-93c3-5ba58c6626c7\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " Oct 29 22:56:16.838050 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.837851 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908e8092-13d9-443e-93c3-5ba58c6626c7-kube-api-access-g8qs4\" (UniqueName: \"kubernetes.io/projected/908e8092-13d9-443e-93c3-5ba58c6626c7-kube-api-access-g8qs4\") pod \"908e8092-13d9-443e-93c3-5ba58c6626c7\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " Oct 29 22:56:16.838050 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.837930 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-model-cache\") pod \"908e8092-13d9-443e-93c3-5ba58c6626c7\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " Oct 29 22:56:16.838050 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.838020 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-dshm\") pod \"908e8092-13d9-443e-93c3-5ba58c6626c7\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " Oct 29 22:56:16.838373 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.838060 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-home\" (UniqueName: \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-home\") pod \"908e8092-13d9-443e-93c3-5ba58c6626c7\" (UID: \"908e8092-13d9-443e-93c3-5ba58c6626c7\") " Oct 29 22:56:16.849834 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.842641 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-home" pod "908e8092-13d9-443e-93c3-5ba58c6626c7" (UID: "908e8092-13d9-443e-93c3-5ba58c6626c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:16.849834 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.846152 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908e8092-13d9-443e-93c3-5ba58c6626c7-tls-certs" pod "908e8092-13d9-443e-93c3-5ba58c6626c7" (UID: "908e8092-13d9-443e-93c3-5ba58c6626c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:56:16.849834 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.846417 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-model-cache" pod "908e8092-13d9-443e-93c3-5ba58c6626c7" (UID: "908e8092-13d9-443e-93c3-5ba58c6626c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:16.864436 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.862602 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-tmp-dir" pod "908e8092-13d9-443e-93c3-5ba58c6626c7" (UID: "908e8092-13d9-443e-93c3-5ba58c6626c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:16.866061 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.866006 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-dshm" pod "908e8092-13d9-443e-93c3-5ba58c6626c7" (UID: "908e8092-13d9-443e-93c3-5ba58c6626c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:16.876848 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.876824 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908e8092-13d9-443e-93c3-5ba58c6626c7-kube-api-access-g8qs4" pod "908e8092-13d9-443e-93c3-5ba58c6626c7" (UID: "908e8092-13d9-443e-93c3-5ba58c6626c7"). InnerVolumeSpecName "kube-api-access-g8qs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:16.939904 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.939873 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-dshm\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:56:16.939904 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.939905 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-home\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:56:16.940115 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.939922 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908e8092-13d9-443e-93c3-5ba58c6626c7-tls-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:56:16.940115 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.939934 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-tmp-dir\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:56:16.940115 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.939947 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8qs4\" (UniqueName: \"kubernetes.io/projected/908e8092-13d9-443e-93c3-5ba58c6626c7-kube-api-access-g8qs4\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:56:16.940115 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.939961 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908e8092-13d9-443e-93c3-5ba58c6626c7-model-cache\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:56:16.946780 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.946708 5057 generic.go:356] "Generic (PLEG): container finished" podID="908e8092-13d9-443e-93c3-5ba58c6626c7" containerID="2f1512582f5a4e085ff16a8554089d2e3f3925805956d0cbbf0555e3e3d75442" exitCode=0 Oct 29 22:56:16.946780 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.946758 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" Oct 29 22:56:16.946984 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.946796 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" event={"ID":"908e8092-13d9-443e-93c3-5ba58c6626c7","Type":"ContainerDied","Data":"2f1512582f5a4e085ff16a8554089d2e3f3925805956d0cbbf0555e3e3d75442"} Oct 29 22:56:16.946984 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.946839 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd" event={"ID":"908e8092-13d9-443e-93c3-5ba58c6626c7","Type":"ContainerDied","Data":"af847851d9aa6c1310298cf84514a4accb613c2cdb97d580bd1833e85d65d487"} Oct 29 22:56:16.946984 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.946860 5057 scope.go:122] "RemoveContainer" containerID="2f1512582f5a4e085ff16a8554089d2e3f3925805956d0cbbf0555e3e3d75442" Oct 29 22:56:16.957707 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.957687 5057 scope.go:122] "RemoveContainer" containerID="2f1512582f5a4e085ff16a8554089d2e3f3925805956d0cbbf0555e3e3d75442" Oct 29 22:56:16.958015 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:56:16.957994 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f1512582f5a4e085ff16a8554089d2e3f3925805956d0cbbf0555e3e3d75442\": container with ID starting with 2f1512582f5a4e085ff16a8554089d2e3f3925805956d0cbbf0555e3e3d75442 not found: ID does not exist" containerID="2f1512582f5a4e085ff16a8554089d2e3f3925805956d0cbbf0555e3e3d75442" Oct 29 22:56:16.958109 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:16.958026 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f1512582f5a4e085ff16a8554089d2e3f3925805956d0cbbf0555e3e3d75442"} err="failed to get container status \"2f1512582f5a4e085ff16a8554089d2e3f3925805956d0cbbf0555e3e3d75442\": rpc error: code = NotFound desc = could not find container \"2f1512582f5a4e085ff16a8554089d2e3f3925805956d0cbbf0555e3e3d75442\": container with ID starting with 2f1512582f5a4e085ff16a8554089d2e3f3925805956d0cbbf0555e3e3d75442 not found: ID does not exist" Oct 29 22:56:17.004963 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:17.004930 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd"] Oct 29 22:56:17.010083 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:17.010055 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-b-03047570-kserve-847c4bb47-rpzzd"] Oct 29 22:56:17.481501 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:17.481468 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="908e8092-13d9-443e-93c3-5ba58c6626c7" path="/var/lib/kubelet/pods/908e8092-13d9-443e-93c3-5ba58c6626c7/volumes" Oct 29 22:56:24.200474 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.200441 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:24.213232 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.213180 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:24.572314 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.572237 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq"] Oct 29 22:56:24.572964 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.572944 5057 cpu_manager.go:414] "containerMap: removing container" podUID="908e8092-13d9-443e-93c3-5ba58c6626c7" containerName="main" Oct 29 22:56:24.572964 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.572963 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908e8092-13d9-443e-93c3-5ba58c6626c7" containerName="main" Oct 29 22:56:24.573119 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.573057 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="908e8092-13d9-443e-93c3-5ba58c6626c7" containerName="main" Oct 29 22:56:24.576443 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.576419 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.614707 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.614505 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq"] Oct 29 22:56:24.634008 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.633967 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d0539b1-a376-4016-900e-382ad30b3e02-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.634236 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.634013 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d0539b1-a376-4016-900e-382ad30b3e02-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.634324 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.634268 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.634387 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.634325 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.634455 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.634404 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.634556 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.634532 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.634658 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.634600 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d0539b1-a376-4016-900e-382ad30b3e02-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.634658 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.634638 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.634765 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.634688 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9j2t\" (UniqueName: \"kubernetes.io/projected/7d0539b1-a376-4016-900e-382ad30b3e02-kube-api-access-c9j2t\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.739208 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.738985 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d0539b1-a376-4016-900e-382ad30b3e02-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.739208 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.739045 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d0539b1-a376-4016-900e-382ad30b3e02-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.739208 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.739114 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.739208 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.739151 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.739208 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.739203 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.739595 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.739261 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.739595 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.739310 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d0539b1-a376-4016-900e-382ad30b3e02-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.739595 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.739336 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.739595 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.739381 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9j2t\" (UniqueName: \"kubernetes.io/projected/7d0539b1-a376-4016-900e-382ad30b3e02-kube-api-access-c9j2t\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.749637 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.745961 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d0539b1-a376-4016-900e-382ad30b3e02-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.749637 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.746273 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.749637 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.746818 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.749637 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.747050 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.749637 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.747525 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.749637 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.747657 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d0539b1-a376-4016-900e-382ad30b3e02-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.779019 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.776219 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9j2t\" (UniqueName: \"kubernetes.io/projected/7d0539b1-a376-4016-900e-382ad30b3e02-kube-api-access-c9j2t\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.779019 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.776646 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.779629 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.779542 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d0539b1-a376-4016-900e-382ad30b3e02-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:24.891251 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:24.891158 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:25.016699 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:25.016664 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:56:25.207889 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:25.207861 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq"] Oct 29 22:56:25.211922 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:56:25.211892 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0539b1_a376_4016_900e_382ad30b3e02.slice/crio-109a03d5784597ae94f91f9b0dcf0058a97a4cdfbf0719650c3f872f7853f59b WatchSource:0}: Error finding container 109a03d5784597ae94f91f9b0dcf0058a97a4cdfbf0719650c3f872f7853f59b: Status 404 returned error can't find the container with id 109a03d5784597ae94f91f9b0dcf0058a97a4cdfbf0719650c3f872f7853f59b Oct 29 22:56:25.214438 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:25.214405 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:56:25.214541 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:25.214488 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:56:25.214621 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:25.214538 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:56:26.017376 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:26.017337 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" event={"ID":"7d0539b1-a376-4016-900e-382ad30b3e02","Type":"ContainerStarted","Data":"d06f6ced558bd847df8ff89d304b0582ec70ae7bcacd5f6afc77241cfdc5562e"} Oct 29 22:56:26.017673 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:26.017649 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" event={"ID":"7d0539b1-a376-4016-900e-382ad30b3e02","Type":"ContainerStarted","Data":"109a03d5784597ae94f91f9b0dcf0058a97a4cdfbf0719650c3f872f7853f59b"} Oct 29 22:56:26.102981 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:26.102928 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" podStartSLOduration=2.102912934 podStartE2EDuration="2.102912934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:26.086546083 +0000 UTC m=+832.738044614" watchObservedRunningTime="2026-10-29 22:56:26.102912934 +0000 UTC m=+832.754411465" Oct 29 22:56:26.892181 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:26.892149 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:26.893813 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:26.893790 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:56:26.893925 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:26.893841 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" podUID="7d0539b1-a376-4016-900e-382ad30b3e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 22:56:27.891585 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:27.891546 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:56:27.891760 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:27.891619 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" podUID="7d0539b1-a376-4016-900e-382ad30b3e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 22:56:28.891943 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:28.891902 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:56:28.892473 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:28.891970 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" podUID="7d0539b1-a376-4016-900e-382ad30b3e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 22:56:29.913213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:29.902518 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:56:29.913213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:29.902581 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" podUID="7d0539b1-a376-4016-900e-382ad30b3e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 22:56:30.321214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:30.320704 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" podUID="f5d589e1-b044-4b23-b722-50c764120c68" containerName="main" containerID="cri-o://8c2bdf4b157b60288ceec0e6b87f3c37aaaa8b9c744cb2eb202cbda2b7925f81" gracePeriod=45 Oct 29 22:56:30.535212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:30.528382 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" podUID="f5d589e1-b044-4b23-b722-50c764120c68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": EOF" Oct 29 22:56:30.773210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:30.770117 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t"] Oct 29 22:56:30.780050 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:30.779369 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:30.982265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:30.830386 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t"] Oct 29 22:56:30.982265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:30.892107 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:56:30.982265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:30.892166 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" podUID="7d0539b1-a376-4016-900e-382ad30b3e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 22:56:30.982265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:30.916293 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db576741-edce-4ad9-8f81-0d8a8f110934-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:30.982265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:30.916383 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4d4w\" (UniqueName: \"kubernetes.io/projected/db576741-edce-4ad9-8f81-0d8a8f110934-kube-api-access-n4d4w\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:30.982265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:30.916419 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:30.982265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:30.916461 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:30.982265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:30.916499 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:30.982265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:30.916533 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db576741-edce-4ad9-8f81-0d8a8f110934-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:30.982265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:30.916654 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:30.982265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:30.916698 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db576741-edce-4ad9-8f81-0d8a8f110934-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:30.982265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:30.916747 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:31.095554 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.071991 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:31.095554 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.072061 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db576741-edce-4ad9-8f81-0d8a8f110934-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:31.095554 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.072098 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:31.095554 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.072149 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db576741-edce-4ad9-8f81-0d8a8f110934-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:31.095554 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.072234 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4d4w\" (UniqueName: \"kubernetes.io/projected/db576741-edce-4ad9-8f81-0d8a8f110934-kube-api-access-n4d4w\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:31.095554 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.072267 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:31.095554 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.072322 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:31.095554 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.072362 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:31.095554 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.072396 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db576741-edce-4ad9-8f81-0d8a8f110934-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:31.170220 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.125361 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:31.170220 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.126630 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db576741-edce-4ad9-8f81-0d8a8f110934-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:31.170220 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.127062 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:31.170220 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.127375 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:31.170220 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.153389 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:31.329249 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.329212 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4d4w\" (UniqueName: \"kubernetes.io/projected/db576741-edce-4ad9-8f81-0d8a8f110934-kube-api-access-n4d4w\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:31.386558 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.338962 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:31.386558 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.339566 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db576741-edce-4ad9-8f81-0d8a8f110934-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:31.386845 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.339948 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db576741-edce-4ad9-8f81-0d8a8f110934-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:31.386845 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.340279 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" podUID="f5d589e1-b044-4b23-b722-50c764120c68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": dial tcp 10.132.0.76:8000: connect: connection refused" Oct 29 22:56:31.386845 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.386774 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:56:31.425212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.412806 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:31.875631 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.875539 5057 generic.go:356] "Generic (PLEG): container finished" podID="f5d589e1-b044-4b23-b722-50c764120c68" containerID="8c2bdf4b157b60288ceec0e6b87f3c37aaaa8b9c744cb2eb202cbda2b7925f81" exitCode=0 Oct 29 22:56:31.876055 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.876025 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" event={"ID":"f5d589e1-b044-4b23-b722-50c764120c68","Type":"ContainerDied","Data":"8c2bdf4b157b60288ceec0e6b87f3c37aaaa8b9c744cb2eb202cbda2b7925f81"} Oct 29 22:56:31.892605 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.892570 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:56:31.892885 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:31.892857 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" podUID="7d0539b1-a376-4016-900e-382ad30b3e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 22:56:32.262220 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.259639 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" podUID="f5d589e1-b044-4b23-b722-50c764120c68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": dial tcp 10.132.0.76:8000: connect: connection refused" Oct 29 22:56:32.720606 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.720577 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:56:32.742680 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.742648 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-tmp-dir\") pod \"f5d589e1-b044-4b23-b722-50c764120c68\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " Oct 29 22:56:32.742680 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.742680 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5d589e1-b044-4b23-b722-50c764120c68-tls-certs\" (UniqueName: \"kubernetes.io/secret/f5d589e1-b044-4b23-b722-50c764120c68-tls-certs\") pod \"f5d589e1-b044-4b23-b722-50c764120c68\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " Oct 29 22:56:32.742857 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.742724 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-model-cache\") pod \"f5d589e1-b044-4b23-b722-50c764120c68\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " Oct 29 22:56:32.742857 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.742769 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-home\" (UniqueName: \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-home\") pod \"f5d589e1-b044-4b23-b722-50c764120c68\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " Oct 29 22:56:32.742857 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.742809 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d589e1-b044-4b23-b722-50c764120c68-kube-api-access-d2d4v\" (UniqueName: \"kubernetes.io/projected/f5d589e1-b044-4b23-b722-50c764120c68-kube-api-access-d2d4v\") pod \"f5d589e1-b044-4b23-b722-50c764120c68\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " Oct 29 22:56:32.742989 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.742963 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-tmp-dir" pod "f5d589e1-b044-4b23-b722-50c764120c68" (UID: "f5d589e1-b044-4b23-b722-50c764120c68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:32.743216 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.743178 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-model-cache" pod "f5d589e1-b044-4b23-b722-50c764120c68" (UID: "f5d589e1-b044-4b23-b722-50c764120c68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:32.743296 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.743214 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-tmp-dir\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:56:32.743296 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.743254 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-home" pod "f5d589e1-b044-4b23-b722-50c764120c68" (UID: "f5d589e1-b044-4b23-b722-50c764120c68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:32.744994 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.744972 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5d589e1-b044-4b23-b722-50c764120c68-tls-certs" pod "f5d589e1-b044-4b23-b722-50c764120c68" (UID: "f5d589e1-b044-4b23-b722-50c764120c68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:56:32.745088 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.745013 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d589e1-b044-4b23-b722-50c764120c68-kube-api-access-d2d4v" pod "f5d589e1-b044-4b23-b722-50c764120c68" (UID: "f5d589e1-b044-4b23-b722-50c764120c68"). InnerVolumeSpecName "kube-api-access-d2d4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:32.759987 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.759959 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t"] Oct 29 22:56:32.763779 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:56:32.763743 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb576741_edce_4ad9_8f81_0d8a8f110934.slice/crio-22e7cbcc1100b07755cac3fa6bf9f10193f622c07a22d8c115672bba5637aac0 WatchSource:0}: Error finding container 22e7cbcc1100b07755cac3fa6bf9f10193f622c07a22d8c115672bba5637aac0: Status 404 returned error can't find the container with id 22e7cbcc1100b07755cac3fa6bf9f10193f622c07a22d8c115672bba5637aac0 Oct 29 22:56:32.765784 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.765751 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:56:32.765872 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.765827 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:56:32.765872 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.765860 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:56:32.843968 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.843942 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-dshm\") pod \"f5d589e1-b044-4b23-b722-50c764120c68\" (UID: \"f5d589e1-b044-4b23-b722-50c764120c68\") " Oct 29 22:56:32.844508 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.844482 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-home\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:56:32.844508 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.844513 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2d4v\" (UniqueName: \"kubernetes.io/projected/f5d589e1-b044-4b23-b722-50c764120c68-kube-api-access-d2d4v\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:56:32.844690 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.844529 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5d589e1-b044-4b23-b722-50c764120c68-tls-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:56:32.844690 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.844544 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-model-cache\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:56:32.846010 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.845985 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-dshm" pod "f5d589e1-b044-4b23-b722-50c764120c68" (UID: "f5d589e1-b044-4b23-b722-50c764120c68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:32.890283 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.886637 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" Oct 29 22:56:32.890283 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.886649 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6" event={"ID":"f5d589e1-b044-4b23-b722-50c764120c68","Type":"ContainerDied","Data":"8fa6f827dedba84f4a4279fce5367ea2ef9a0a5987d92897ef09b481f0d80f24"} Oct 29 22:56:32.890283 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.886706 5057 scope.go:122] "RemoveContainer" containerID="8c2bdf4b157b60288ceec0e6b87f3c37aaaa8b9c744cb2eb202cbda2b7925f81" Oct 29 22:56:32.890283 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.888837 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" event={"ID":"db576741-edce-4ad9-8f81-0d8a8f110934","Type":"ContainerStarted","Data":"89f43b34408003a349700c605cc515a4cf275d747478e6c10ce45f3c8d65b6d0"} Oct 29 22:56:32.890283 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.888889 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" event={"ID":"db576741-edce-4ad9-8f81-0d8a8f110934","Type":"ContainerStarted","Data":"22e7cbcc1100b07755cac3fa6bf9f10193f622c07a22d8c115672bba5637aac0"} Oct 29 22:56:32.892118 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.892089 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:56:32.892251 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.892131 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" podUID="7d0539b1-a376-4016-900e-382ad30b3e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 22:56:32.945010 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.944970 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6"] Oct 29 22:56:32.945434 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.945412 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5d589e1-b044-4b23-b722-50c764120c68-dshm\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:56:32.953287 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:32.953263 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/e2e-conflict-a-03047570-kserve-78b89947b5-k2zh6"] Oct 29 22:56:33.001737 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:33.001669 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" podStartSLOduration=3.001649495 podStartE2EDuration="3.001649495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:32.99839657 +0000 UTC m=+839.649895100" watchObservedRunningTime="2026-10-29 22:56:33.001649495 +0000 UTC m=+839.653148027" Oct 29 22:56:33.415203 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:33.415157 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:33.417109 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:33.417081 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:56:33.417272 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:33.417129 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" podUID="db576741-edce-4ad9-8f81-0d8a8f110934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 29 22:56:33.490456 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:33.490119 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5d589e1-b044-4b23-b722-50c764120c68" path="/var/lib/kubelet/pods/f5d589e1-b044-4b23-b722-50c764120c68/volumes" Oct 29 22:56:33.944040 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:56:33.943902 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:56:34.009936 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:34.009902 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:34.011143 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:34.011117 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:34.012011 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:34.011987 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 22:56:34.415520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:34.415441 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:56:34.415520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:34.415497 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" podUID="db576741-edce-4ad9-8f81-0d8a8f110934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 29 22:56:35.414621 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:35.414574 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:56:35.414803 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:35.414641 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" podUID="db576741-edce-4ad9-8f81-0d8a8f110934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 29 22:56:36.415544 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:36.415507 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:56:36.416018 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:36.415568 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" podUID="db576741-edce-4ad9-8f81-0d8a8f110934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 29 22:56:37.414671 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:37.414624 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:56:37.414867 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:37.414684 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" podUID="db576741-edce-4ad9-8f81-0d8a8f110934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 29 22:56:38.415459 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:38.415421 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:56:38.415840 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:38.415469 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" podUID="db576741-edce-4ad9-8f81-0d8a8f110934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 29 22:56:39.415179 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:39.415142 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:56:39.415364 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:39.415230 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" podUID="db576741-edce-4ad9-8f81-0d8a8f110934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 29 22:56:40.419615 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:40.419584 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:40.420050 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:40.420027 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:40.420837 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:40.420806 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 22:56:44.040430 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.040392 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:56:44.535090 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.535058 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7"] Oct 29 22:56:44.536266 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.536236 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f5d589e1-b044-4b23-b722-50c764120c68" containerName="main" Oct 29 22:56:44.536266 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.536264 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5d589e1-b044-4b23-b722-50c764120c68" containerName="main" Oct 29 22:56:44.536455 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.536427 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5d589e1-b044-4b23-b722-50c764120c68" containerName="main" Oct 29 22:56:44.540547 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.540505 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:44.559005 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.558979 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7"] Oct 29 22:56:44.568024 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.567997 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e57216\"/\"openshift-service-ca.crt\"" Oct 29 22:56:44.568129 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.568088 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e57216\"/\"kube-root-ca.crt\"" Oct 29 22:56:44.622452 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.622408 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e57216\"/\"default-dockercfg-6ltmm\"" Oct 29 22:56:44.622665 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.622456 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e57216\"/\"distinct-w0-e57216-kserve-self-signed-certs\"" Oct 29 22:56:44.679824 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.679775 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-model-cache\") pod \"distinct-w0-e57216-kserve-5f7fc89c57-nd7l7\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:44.680001 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.679830 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hls9g\" (UniqueName: \"kubernetes.io/projected/5eee59f4-800b-4a65-a111-f11fe3afdafe-kube-api-access-hls9g\") pod \"distinct-w0-e57216-kserve-5f7fc89c57-nd7l7\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:44.680001 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.679869 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-home\") pod \"distinct-w0-e57216-kserve-5f7fc89c57-nd7l7\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:44.680001 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.679961 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eee59f4-800b-4a65-a111-f11fe3afdafe-tls-certs\") pod \"distinct-w0-e57216-kserve-5f7fc89c57-nd7l7\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:44.680179 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.680145 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-dshm\") pod \"distinct-w0-e57216-kserve-5f7fc89c57-nd7l7\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:44.680252 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.680224 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-tmp-dir\") pod \"distinct-w0-e57216-kserve-5f7fc89c57-nd7l7\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:44.781979 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.781936 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-dshm\") pod \"distinct-w0-e57216-kserve-5f7fc89c57-nd7l7\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:44.782282 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.782257 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-tmp-dir\") pod \"distinct-w0-e57216-kserve-5f7fc89c57-nd7l7\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:44.782464 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.782449 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-model-cache\") pod \"distinct-w0-e57216-kserve-5f7fc89c57-nd7l7\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:44.782567 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.782552 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hls9g\" (UniqueName: \"kubernetes.io/projected/5eee59f4-800b-4a65-a111-f11fe3afdafe-kube-api-access-hls9g\") pod \"distinct-w0-e57216-kserve-5f7fc89c57-nd7l7\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:44.782782 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.782764 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-home\") pod \"distinct-w0-e57216-kserve-5f7fc89c57-nd7l7\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:44.782912 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.782897 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eee59f4-800b-4a65-a111-f11fe3afdafe-tls-certs\") pod \"distinct-w0-e57216-kserve-5f7fc89c57-nd7l7\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:44.788676 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.788615 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-model-cache\") pod \"distinct-w0-e57216-kserve-5f7fc89c57-nd7l7\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:44.802228 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.798550 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-tmp-dir\") pod \"distinct-w0-e57216-kserve-5f7fc89c57-nd7l7\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:44.802228 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.799106 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-home\") pod \"distinct-w0-e57216-kserve-5f7fc89c57-nd7l7\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:44.802228 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.799423 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-dshm\") pod \"distinct-w0-e57216-kserve-5f7fc89c57-nd7l7\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:44.802228 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.799556 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eee59f4-800b-4a65-a111-f11fe3afdafe-tls-certs\") pod \"distinct-w0-e57216-kserve-5f7fc89c57-nd7l7\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:44.827231 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.827136 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hls9g\" (UniqueName: \"kubernetes.io/projected/5eee59f4-800b-4a65-a111-f11fe3afdafe-kube-api-access-hls9g\") pod \"distinct-w0-e57216-kserve-5f7fc89c57-nd7l7\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:44.873137 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:44.873103 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:45.227518 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:56:45.223153 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5eee59f4_800b_4a65_a111_f11fe3afdafe.slice/crio-8cd158039bc8211cb306d61202a0639849fe7d526b8ff63ea76482e7c8c26541 WatchSource:0}: Error finding container 8cd158039bc8211cb306d61202a0639849fe7d526b8ff63ea76482e7c8c26541: Status 404 returned error can't find the container with id 8cd158039bc8211cb306d61202a0639849fe7d526b8ff63ea76482e7c8c26541 Oct 29 22:56:45.227518 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:45.226286 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7"] Oct 29 22:56:46.026972 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:46.026886 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" event={"ID":"5eee59f4-800b-4a65-a111-f11fe3afdafe","Type":"ContainerStarted","Data":"5ab12b8f8708b4a4f9e9087744f034ee4b6c5d98083ed3f661d2acd0d2fc76de"} Oct 29 22:56:46.026972 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:46.026939 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" event={"ID":"5eee59f4-800b-4a65-a111-f11fe3afdafe","Type":"ContainerStarted","Data":"8cd158039bc8211cb306d61202a0639849fe7d526b8ff63ea76482e7c8c26541"} Oct 29 22:56:46.072849 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:46.072788 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podStartSLOduration=2.07277047 podStartE2EDuration="2.07277047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:46.072589843 +0000 UTC m=+852.724088376" watchObservedRunningTime="2026-10-29 22:56:46.07277047 +0000 UTC m=+852.724269001" Oct 29 22:56:46.873462 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:46.873423 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:51.715078 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.715039 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg"] Oct 29 22:56:51.719201 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.719165 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.733596 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.733560 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.733733 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.733638 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.733733 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.733671 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.733812 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.733777 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.733812 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.733801 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pngrj\" (UniqueName: \"kubernetes.io/projected/531e93a3-f8f2-4b85-966c-e1830958da9f-kube-api-access-pngrj\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.733893 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.733835 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/531e93a3-f8f2-4b85-966c-e1830958da9f-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.734004 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.733984 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.734047 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.734023 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.734094 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.734061 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.767961 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.767929 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg"] Oct 29 22:56:51.835085 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.835054 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.835297 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.835096 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pngrj\" (UniqueName: \"kubernetes.io/projected/531e93a3-f8f2-4b85-966c-e1830958da9f-kube-api-access-pngrj\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.835297 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.835130 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/531e93a3-f8f2-4b85-966c-e1830958da9f-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.835297 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.835175 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.835297 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.835217 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.835297 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.835256 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.835554 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.835372 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.835554 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.835402 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.835554 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.835434 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.835925 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.835703 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.835925 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.835823 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.836089 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.836022 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.836131 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.836101 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.836260 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.836230 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/531e93a3-f8f2-4b85-966c-e1830958da9f-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.838353 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.838332 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.838623 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.838598 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.860721 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.860695 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:51.860812 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:51.860757 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pngrj\" (UniqueName: \"kubernetes.io/projected/531e93a3-f8f2-4b85-966c-e1830958da9f-kube-api-access-pngrj\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:52.034666 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:52.034560 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:52.372264 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:52.363998 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg"] Oct 29 22:56:52.423799 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:56:52.423762 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531e93a3_f8f2_4b85_966c_e1830958da9f.slice/crio-c888ff78a40f1e69723dff3fe0e63a7ac7049948d1eccd67bcbdbfe6197729e9 WatchSource:0}: Error finding container c888ff78a40f1e69723dff3fe0e63a7ac7049948d1eccd67bcbdbfe6197729e9: Status 404 returned error can't find the container with id c888ff78a40f1e69723dff3fe0e63a7ac7049948d1eccd67bcbdbfe6197729e9 Oct 29 22:56:52.426590 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:52.426557 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:56:52.426756 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:52.426739 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:56:52.426843 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:52.426831 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:56:53.086441 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:53.086401 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" event={"ID":"531e93a3-f8f2-4b85-966c-e1830958da9f","Type":"ContainerStarted","Data":"d4cb6aa44d50c4e8a1792b169137918cb6a3a9c992b54ac4abc25fcdaffb63a9"} Oct 29 22:56:53.086847 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:53.086471 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" event={"ID":"531e93a3-f8f2-4b85-966c-e1830958da9f","Type":"ContainerStarted","Data":"c888ff78a40f1e69723dff3fe0e63a7ac7049948d1eccd67bcbdbfe6197729e9"} Oct 29 22:56:53.155264 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:53.155174 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" podStartSLOduration=2.155153387 podStartE2EDuration="2.155153387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:53.154490777 +0000 UTC m=+859.805989309" watchObservedRunningTime="2026-10-29 22:56:53.155153387 +0000 UTC m=+859.806651918" Oct 29 22:56:54.036554 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:54.036526 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:56:54.041316 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:54.041287 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:56:54.041513 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:54.041482 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" podUID="531e93a3-f8f2-4b85-966c-e1830958da9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 22:56:54.873414 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:54.873381 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:54.886801 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:54.886775 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:55.035662 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:55.035625 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:56:55.035928 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:55.035901 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" podUID="531e93a3-f8f2-4b85-966c-e1830958da9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 22:56:55.158769 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:55.158690 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:56:56.047210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:56.040701 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:56:56.047210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:56.040772 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" podUID="531e93a3-f8f2-4b85-966c-e1830958da9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 22:56:57.038682 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:57.038642 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:56:57.039003 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:57.038973 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" podUID="531e93a3-f8f2-4b85-966c-e1830958da9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 22:56:58.037624 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.037584 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:56:58.038238 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.038202 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" podUID="531e93a3-f8f2-4b85-966c-e1830958da9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 22:56:58.158569 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.158525 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj"] Oct 29 22:56:58.165731 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.165695 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.191248 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.191211 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj"] Oct 29 22:56:58.261212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.245953 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.261212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.246017 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.261212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.246049 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.261212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.246099 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.261212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.246133 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.261212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.246171 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.261212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.246279 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.261212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.246356 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9kbv\" (UniqueName: \"kubernetes.io/projected/ed5d8696-66e9-4e1d-9419-2597fef50ec7-kube-api-access-r9kbv\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.261212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.246439 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.347290 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.347195 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.347465 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.347295 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.347465 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.347320 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.347465 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.347343 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.347465 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.347385 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.347465 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.347416 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.347465 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.347449 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.347799 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.347537 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.347799 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.347607 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9kbv\" (UniqueName: \"kubernetes.io/projected/ed5d8696-66e9-4e1d-9419-2597fef50ec7-kube-api-access-r9kbv\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.347799 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.347610 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.348032 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.348007 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.348092 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.348031 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.348322 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.348302 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.348691 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.348667 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.351531 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.351476 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.352102 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.352078 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.377212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.377169 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.377859 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.377833 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9kbv\" (UniqueName: \"kubernetes.io/projected/ed5d8696-66e9-4e1d-9419-2597fef50ec7-kube-api-access-r9kbv\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.480609 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.480576 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:58.805616 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:56:58.805584 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5d8696_66e9_4e1d_9419_2597fef50ec7.slice/crio-c26b7b58392af387b8141ade178425f3f8b439d471935fb953dab455e545b2a7 WatchSource:0}: Error finding container c26b7b58392af387b8141ade178425f3f8b439d471935fb953dab455e545b2a7: Status 404 returned error can't find the container with id c26b7b58392af387b8141ade178425f3f8b439d471935fb953dab455e545b2a7 Oct 29 22:56:58.806691 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.806664 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj"] Oct 29 22:56:58.808140 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.808106 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:56:58.808246 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.808170 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:56:58.808246 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:58.808236 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:56:59.035531 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:59.035492 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:56:59.035729 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:59.035568 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" podUID="531e93a3-f8f2-4b85-966c-e1830958da9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 22:56:59.181562 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:59.181530 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" event={"ID":"ed5d8696-66e9-4e1d-9419-2597fef50ec7","Type":"ContainerStarted","Data":"5bbef6450b52a2881cecb2cd8432e4ef841f31ac3722466986bc1f46107a55bf"} Oct 29 22:56:59.181957 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:59.181584 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" event={"ID":"ed5d8696-66e9-4e1d-9419-2597fef50ec7","Type":"ContainerStarted","Data":"c26b7b58392af387b8141ade178425f3f8b439d471935fb953dab455e545b2a7"} Oct 29 22:56:59.231829 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:59.231770 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" podStartSLOduration=1.231747511 podStartE2EDuration="1.231747511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:59.231500176 +0000 UTC m=+865.882998706" watchObservedRunningTime="2026-10-29 22:56:59.231747511 +0000 UTC m=+865.883246042" Oct 29 22:56:59.481788 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:59.481696 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:56:59.483646 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:59.483613 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:56:59.483799 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:56:59.483672 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" podUID="ed5d8696-66e9-4e1d-9419-2597fef50ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:57:00.055332 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:00.053888 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:57:00.055332 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:00.055149 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:57:00.056002 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:00.055984 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 22:57:00.481412 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:00.481377 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:00.481798 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:00.481440 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" podUID="ed5d8696-66e9-4e1d-9419-2597fef50ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:57:01.481656 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:01.481622 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:01.482115 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:01.481678 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" podUID="ed5d8696-66e9-4e1d-9419-2597fef50ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:57:02.481112 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:02.481076 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:02.481327 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:02.481127 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" podUID="ed5d8696-66e9-4e1d-9419-2597fef50ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:57:03.009207 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.008247 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l"] Oct 29 22:57:03.081629 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.081593 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l"] Oct 29 22:57:03.081944 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.081927 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:03.143207 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.127858 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-99dfb0\"/\"openshift-service-ca.crt\"" Oct 29 22:57:03.143207 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.127956 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-99dfb0\"/\"default-dockercfg-nf4k7\"" Oct 29 22:57:03.143207 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.128597 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-99dfb0\"/\"distinct-2-w1-99dfb0-kserve-self-signed-certs\"" Oct 29 22:57:03.143207 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.128656 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-99dfb0\"/\"kube-root-ca.crt\"" Oct 29 22:57:03.252235 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.243368 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-tmp-dir\") pod \"distinct-2-w1-99dfb0-kserve-75f949566f-znq7l\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:03.300492 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.300406 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-home\") pod \"distinct-2-w1-99dfb0-kserve-75f949566f-znq7l\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:03.300692 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.300667 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-dshm\") pod \"distinct-2-w1-99dfb0-kserve-75f949566f-znq7l\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:03.300765 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.300733 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-model-cache\") pod \"distinct-2-w1-99dfb0-kserve-75f949566f-znq7l\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:03.301019 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.300992 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a40c346b-32d3-45d0-959f-76a53fc1083d-tls-certs\") pod \"distinct-2-w1-99dfb0-kserve-75f949566f-znq7l\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:03.314209 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.301088 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fssgp\" (UniqueName: \"kubernetes.io/projected/a40c346b-32d3-45d0-959f-76a53fc1083d-kube-api-access-fssgp\") pod \"distinct-2-w1-99dfb0-kserve-75f949566f-znq7l\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:03.417212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.412593 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-dshm\") pod \"distinct-2-w1-99dfb0-kserve-75f949566f-znq7l\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:03.417212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.412669 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-model-cache\") pod \"distinct-2-w1-99dfb0-kserve-75f949566f-znq7l\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:03.417212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.412743 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a40c346b-32d3-45d0-959f-76a53fc1083d-tls-certs\") pod \"distinct-2-w1-99dfb0-kserve-75f949566f-znq7l\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:03.417212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.412815 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fssgp\" (UniqueName: \"kubernetes.io/projected/a40c346b-32d3-45d0-959f-76a53fc1083d-kube-api-access-fssgp\") pod \"distinct-2-w1-99dfb0-kserve-75f949566f-znq7l\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:03.417212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.412972 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-tmp-dir\") pod \"distinct-2-w1-99dfb0-kserve-75f949566f-znq7l\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:03.417212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.413067 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-home\") pod \"distinct-2-w1-99dfb0-kserve-75f949566f-znq7l\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:03.417212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.413683 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-home\") pod \"distinct-2-w1-99dfb0-kserve-75f949566f-znq7l\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:03.430507 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.428707 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-model-cache\") pod \"distinct-2-w1-99dfb0-kserve-75f949566f-znq7l\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:03.435447 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.435415 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a40c346b-32d3-45d0-959f-76a53fc1083d-tls-certs\") pod \"distinct-2-w1-99dfb0-kserve-75f949566f-znq7l\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:03.436293 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.436264 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-tmp-dir\") pod \"distinct-2-w1-99dfb0-kserve-75f949566f-znq7l\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:03.436641 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.436622 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-dshm\") pod \"distinct-2-w1-99dfb0-kserve-75f949566f-znq7l\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:03.510029 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.509994 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:03.510548 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.510401 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" podUID="ed5d8696-66e9-4e1d-9419-2597fef50ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:57:03.540210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.513575 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fssgp\" (UniqueName: \"kubernetes.io/projected/a40c346b-32d3-45d0-959f-76a53fc1083d-kube-api-access-fssgp\") pod \"distinct-2-w1-99dfb0-kserve-75f949566f-znq7l\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:03.574101 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:03.574017 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:04.561247 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:57:04.561204 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:57:04.586261 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:04.563357 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:04.599330 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:04.599289 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" podUID="ed5d8696-66e9-4e1d-9419-2597fef50ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:57:04.708211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:04.698734 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j"] Oct 29 22:57:04.964587 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:04.964502 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.050378 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.050332 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.075404 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.075355 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bswd\" (UniqueName: \"kubernetes.io/projected/5ac55fce-ddaa-4ffc-902d-07a374f6d967-kube-api-access-6bswd\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.075776 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.075749 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.100334 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.100286 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.100621 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.100599 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.100789 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.100771 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.121625 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.121577 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.121953 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.121930 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.122093 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.122076 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.203330 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.102116 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 22:57:05.203330 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.156611 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:05.203330 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.171262 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j"] Oct 29 22:57:05.212291 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.212239 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded" Oct 29 22:57:05.287300 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.246041 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.287300 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.246120 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.287300 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.246168 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.287300 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.246245 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.287300 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.246284 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.287300 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.246359 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.287300 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.246438 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bswd\" (UniqueName: \"kubernetes.io/projected/5ac55fce-ddaa-4ffc-902d-07a374f6d967-kube-api-access-6bswd\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.287300 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.246529 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.287300 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.246558 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.287300 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.274505 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.288481 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.288452 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:05.288654 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.288626 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" podUID="7d0539b1-a376-4016-900e-382ad30b3e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:05.289771 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.289747 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l"] Oct 29 22:57:05.289976 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.289947 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.302218 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.291065 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.302485 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.293754 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.302635 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.293965 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.485086 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.485052 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.526320 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.508406 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:05.526320 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.508465 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" podUID="ed5d8696-66e9-4e1d-9419-2597fef50ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:57:05.556298 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.544041 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.587225 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.557121 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.891267 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.882330 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bswd\" (UniqueName: \"kubernetes.io/projected/5ac55fce-ddaa-4ffc-902d-07a374f6d967-kube-api-access-6bswd\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:05.952209 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.944788 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:05.990796 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:05.990742 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:06.113078 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:06.113047 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:06.158233 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:06.158118 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:06.158866 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:06.158828 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 22:57:06.495777 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:06.495730 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:06.496087 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:06.496052 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" podUID="ed5d8696-66e9-4e1d-9419-2597fef50ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:57:06.780257 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:57:06.776990 5057 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.288s" Oct 29 22:57:06.860334 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:06.860290 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" event={"ID":"a40c346b-32d3-45d0-959f-76a53fc1083d","Type":"ContainerStarted","Data":"11137751ef519de163c8db39cbc9252b4471b167c06ad3195707ddeb76cc2aa5"} Oct 29 22:57:06.887372 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:06.887339 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:57:06.887661 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:06.887642 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:57:06.948636 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:06.948443 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:06.949604 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:06.949581 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:57:07.206422 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:07.206377 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 22:57:07.210313 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:07.210236 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:07.714439 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:07.674384 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:07.714439 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:07.674445 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" podUID="ed5d8696-66e9-4e1d-9419-2597fef50ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:57:07.795797 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:07.795764 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:57:07.813095 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:07.812960 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:57:07.824961 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:07.819654 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:57:08.068250 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:08.068218 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j"] Oct 29 22:57:08.071048 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:57:08.071018 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac55fce_ddaa_4ffc_902d_07a374f6d967.slice/crio-83b86b0d2cad2b4f514ece83f52207780f39a14649b322532670070bf68ffe45 WatchSource:0}: Error finding container 83b86b0d2cad2b4f514ece83f52207780f39a14649b322532670070bf68ffe45: Status 404 returned error can't find the container with id 83b86b0d2cad2b4f514ece83f52207780f39a14649b322532670070bf68ffe45 Oct 29 22:57:08.076178 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:08.074085 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:08.076178 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:08.074324 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:08.076178 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:08.074413 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:08.481229 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:08.481170 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:08.481420 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:08.481255 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" podUID="ed5d8696-66e9-4e1d-9419-2597fef50ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:57:08.877246 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:08.877115 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" event={"ID":"a40c346b-32d3-45d0-959f-76a53fc1083d","Type":"ContainerStarted","Data":"97046d49c1fab966113e523587494209f22d35e85bb883299cf0ec779d08fa5d"} Oct 29 22:57:08.878909 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:08.878886 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" event={"ID":"5ac55fce-ddaa-4ffc-902d-07a374f6d967","Type":"ContainerStarted","Data":"f91b03e21921908c78edd1c4fb4eac34c120b38a8dfedf5ace832ad296518175"} Oct 29 22:57:08.879022 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:08.878930 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" event={"ID":"5ac55fce-ddaa-4ffc-902d-07a374f6d967","Type":"ContainerStarted","Data":"83b86b0d2cad2b4f514ece83f52207780f39a14649b322532670070bf68ffe45"} Oct 29 22:57:08.923119 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:08.923052 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podStartSLOduration=6.923032433 podStartE2EDuration="6.923032433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:08.921588273 +0000 UTC m=+875.573086885" watchObservedRunningTime="2026-10-29 22:57:08.923032433 +0000 UTC m=+875.574530966" Oct 29 22:57:08.981052 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:08.980985 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" podStartSLOduration=4.98096582 podStartE2EDuration="4.98096582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:08.976655585 +0000 UTC m=+875.628154117" watchObservedRunningTime="2026-10-29 22:57:08.98096582 +0000 UTC m=+875.632464351" Oct 29 22:57:09.118297 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:09.118261 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:09.118696 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:09.118668 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 29 22:57:09.118873 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:09.118848 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" podUID="5ac55fce-ddaa-4ffc-902d-07a374f6d967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 29 22:57:09.569228 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:09.567573 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:57:09.569228 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:09.568447 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:57:09.569724 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:09.569702 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 22:57:09.574446 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:09.574426 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:10.114381 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:10.114347 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 29 22:57:10.114778 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:10.114402 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" podUID="5ac55fce-ddaa-4ffc-902d-07a374f6d967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 29 22:57:11.114751 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:11.114717 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 29 22:57:11.115255 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:11.115229 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" podUID="5ac55fce-ddaa-4ffc-902d-07a374f6d967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 29 22:57:12.113739 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:12.113704 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 29 22:57:12.113892 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:12.113756 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" podUID="5ac55fce-ddaa-4ffc-902d-07a374f6d967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 29 22:57:13.114129 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.114096 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 29 22:57:13.114682 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.114152 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" podUID="5ac55fce-ddaa-4ffc-902d-07a374f6d967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 29 22:57:13.197253 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.197222 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7"] Oct 29 22:57:13.201685 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.201660 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.266097 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.266064 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7"] Oct 29 22:57:13.287681 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.287640 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.287827 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.287689 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.287987 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.287956 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.288098 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.288055 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.288179 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.288120 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljvzr\" (UniqueName: \"kubernetes.io/projected/d205536f-2f06-4f8a-9ef1-e640f0904f6c-kube-api-access-ljvzr\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.288463 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.288439 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.288656 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.288633 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.288774 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.288756 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.288891 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.288833 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.390392 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.390299 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.390392 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.390340 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.390392 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.390389 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.390611 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.390406 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.390611 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.390429 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljvzr\" (UniqueName: \"kubernetes.io/projected/d205536f-2f06-4f8a-9ef1-e640f0904f6c-kube-api-access-ljvzr\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.390611 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.390487 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.390611 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.390504 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.390611 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.390555 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.390611 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.390571 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.390842 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.390700 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.390842 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.390804 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.391212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.391169 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.391485 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.391465 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.391769 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.391747 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.393049 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.393025 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.393121 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.393058 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.415907 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.415884 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.416302 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.416282 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljvzr\" (UniqueName: \"kubernetes.io/projected/d205536f-2f06-4f8a-9ef1-e640f0904f6c-kube-api-access-ljvzr\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.515974 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.515938 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:13.588459 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.587024 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:13.603614 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.603348 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:13.881116 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:57:13.880652 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd205536f_2f06_4f8a_9ef1_e640f0904f6c.slice/crio-77bfe2b7a7994b7d8abc25b77a885ce283369b5316c9cd97f1b589cbe1e27711 WatchSource:0}: Error finding container 77bfe2b7a7994b7d8abc25b77a885ce283369b5316c9cd97f1b589cbe1e27711: Status 404 returned error can't find the container with id 77bfe2b7a7994b7d8abc25b77a885ce283369b5316c9cd97f1b589cbe1e27711 Oct 29 22:57:13.883221 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.883163 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:13.883325 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.883246 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:13.883325 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.883278 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:13.884453 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.884425 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7"] Oct 29 22:57:13.967531 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.966216 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" event={"ID":"d205536f-2f06-4f8a-9ef1-e640f0904f6c","Type":"ContainerStarted","Data":"77bfe2b7a7994b7d8abc25b77a885ce283369b5316c9cd97f1b589cbe1e27711"} Oct 29 22:57:13.981287 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:13.981260 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:14.114593 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:14.114551 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 29 22:57:14.114954 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:14.114622 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" podUID="5ac55fce-ddaa-4ffc-902d-07a374f6d967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 29 22:57:14.975040 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:14.975006 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" event={"ID":"d205536f-2f06-4f8a-9ef1-e640f0904f6c","Type":"ContainerStarted","Data":"565e7676cf8de179628f79102d9f0f0b9fda2fbeac48cfcb067a103aef2dc4ce"} Oct 29 22:57:15.114420 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:15.114381 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 29 22:57:15.114591 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:15.114431 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" podUID="5ac55fce-ddaa-4ffc-902d-07a374f6d967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 29 22:57:15.517019 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:15.516979 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:15.519220 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:15.519170 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:57:15.519323 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:15.519261 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" podUID="d205536f-2f06-4f8a-9ef1-e640f0904f6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 22:57:16.505606 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:16.505569 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:16.506863 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:16.506748 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:16.510951 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:16.510926 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 22:57:16.517401 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:16.517372 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:57:16.517849 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:16.517424 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" podUID="d205536f-2f06-4f8a-9ef1-e640f0904f6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 22:57:16.571182 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:16.571128 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" podStartSLOduration=3.571113927 podStartE2EDuration="3.571113927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:15.066180617 +0000 UTC m=+881.717679147" watchObservedRunningTime="2026-10-29 22:57:16.571113927 +0000 UTC m=+883.222612457" Oct 29 22:57:17.053680 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:17.053633 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2"] Oct 29 22:57:17.156413 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:17.152401 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2"] Oct 29 22:57:17.185352 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:17.152634 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2" Oct 29 22:57:17.190630 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:17.184030 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56xgd\" (UniqueName: \"kubernetes.io/projected/0793f543-6d14-4794-934a-26ea2d40e730-kube-api-access-56xgd\") pod \"payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2\" (UID: \"0793f543-6d14-4794-934a-26ea2d40e730\") " pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2" Oct 29 22:57:17.190630 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:17.190496 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0793f543-6d14-4794-934a-26ea2d40e730-plugins-config-volume\") pod \"payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2\" (UID: \"0793f543-6d14-4794-934a-26ea2d40e730\") " pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2" Oct 29 22:57:17.207116 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:17.207092 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-79e6c631\"" Oct 29 22:57:17.220695 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:17.220664 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-79e6c631-dockercfg-7cbf6\"" Oct 29 22:57:17.292131 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:17.292072 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56xgd\" (UniqueName: \"kubernetes.io/projected/0793f543-6d14-4794-934a-26ea2d40e730-kube-api-access-56xgd\") pod \"payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2\" (UID: \"0793f543-6d14-4794-934a-26ea2d40e730\") " pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2" Oct 29 22:57:17.292541 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:17.292512 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0793f543-6d14-4794-934a-26ea2d40e730-plugins-config-volume\") pod \"payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2\" (UID: \"0793f543-6d14-4794-934a-26ea2d40e730\") " pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2" Oct 29 22:57:17.293267 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:17.293246 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0793f543-6d14-4794-934a-26ea2d40e730-plugins-config-volume\") pod \"payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2\" (UID: \"0793f543-6d14-4794-934a-26ea2d40e730\") " pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2" Oct 29 22:57:17.317518 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:17.317452 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56xgd\" (UniqueName: \"kubernetes.io/projected/0793f543-6d14-4794-934a-26ea2d40e730-kube-api-access-56xgd\") pod \"payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2\" (UID: \"0793f543-6d14-4794-934a-26ea2d40e730\") " pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2" Oct 29 22:57:17.509092 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:17.509058 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2" Oct 29 22:57:17.516428 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:17.516399 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:57:17.516553 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:17.516443 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" podUID="d205536f-2f06-4f8a-9ef1-e640f0904f6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 22:57:17.865818 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:17.864796 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2"] Oct 29 22:57:18.204427 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:18.204391 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2" event={"ID":"0793f543-6d14-4794-934a-26ea2d40e730","Type":"ContainerStarted","Data":"7c238644441ecd6ec079329487737109811ef5f1829fac3a335f3f28cebc4e7d"} Oct 29 22:57:18.204624 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:18.204448 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2" event={"ID":"0793f543-6d14-4794-934a-26ea2d40e730","Type":"ContainerStarted","Data":"962df39aed95e3ba812028ba1beca1526c1b1108ae2b5524a59e4069c9b4f86f"} Oct 29 22:57:18.204624 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:18.204482 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2" Oct 29 22:57:18.259751 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:18.259688 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2" podStartSLOduration=2.2596678470000002 podStartE2EDuration="2.259667847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:18.257357258 +0000 UTC m=+884.908855788" watchObservedRunningTime="2026-10-29 22:57:18.259667847 +0000 UTC m=+884.911166378" Oct 29 22:57:18.517073 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:18.516982 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:57:18.517073 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:18.517042 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" podUID="d205536f-2f06-4f8a-9ef1-e640f0904f6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 22:57:19.517106 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:19.517066 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:57:19.517505 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:19.517132 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" podUID="d205536f-2f06-4f8a-9ef1-e640f0904f6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 22:57:20.517684 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:20.517112 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:57:20.517684 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:20.517223 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" podUID="d205536f-2f06-4f8a-9ef1-e640f0904f6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 22:57:21.621507 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:21.621473 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:21.622016 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:21.621812 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:21.623153 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:21.623133 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 22:57:23.291491 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.291454 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps"] Oct 29 22:57:23.295756 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.295734 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.330558 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.330525 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e57216-openshift-default-dockercfg-jh4hn\"" Oct 29 22:57:23.335550 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.335507 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps"] Oct 29 22:57:23.432930 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.432893 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.433129 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.433026 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.433129 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.433062 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.433129 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.433079 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.433310 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.433160 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g97wl\" (UniqueName: \"kubernetes.io/projected/9a4615b1-ff3d-4339-853a-f5d5247a5573-kube-api-access-g97wl\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.433507 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.433485 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.433566 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.433518 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a4615b1-ff3d-4339-853a-f5d5247a5573-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.433623 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.433577 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.433680 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.433624 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.535169 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.535128 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.535408 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.535179 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.535408 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.535290 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g97wl\" (UniqueName: \"kubernetes.io/projected/9a4615b1-ff3d-4339-853a-f5d5247a5573-kube-api-access-g97wl\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.535408 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.535349 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.535408 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.535376 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a4615b1-ff3d-4339-853a-f5d5247a5573-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.535688 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.535424 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.535688 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.535485 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.535688 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.535578 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.535688 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.535684 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.535831 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.535779 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.535868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.535852 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.535965 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.535941 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.536044 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.535992 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.536574 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.536553 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a4615b1-ff3d-4339-853a-f5d5247a5573-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.537775 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.537749 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.538200 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.538167 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.559854 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.559800 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.560233 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.560170 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g97wl\" (UniqueName: \"kubernetes.io/projected/9a4615b1-ff3d-4339-853a-f5d5247a5573-kube-api-access-g97wl\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.619049 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.614241 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:23.983557 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:57:23.983525 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4615b1_ff3d_4339_853a_f5d5247a5573.slice/crio-a12fb4555f10562286caac8eb77e90e5213b66abb1fe7466eb5002a166a6eea4 WatchSource:0}: Error finding container a12fb4555f10562286caac8eb77e90e5213b66abb1fe7466eb5002a166a6eea4: Status 404 returned error can't find the container with id a12fb4555f10562286caac8eb77e90e5213b66abb1fe7466eb5002a166a6eea4 Oct 29 22:57:23.983892 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.983869 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps"] Oct 29 22:57:23.985992 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.985962 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:23.986067 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.986020 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:23.986067 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:23.986051 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:24.256250 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:24.256120 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" event={"ID":"9a4615b1-ff3d-4339-853a-f5d5247a5573","Type":"ContainerStarted","Data":"0957fa92c5d38be4195b35dc646131ab9e551749a298ff80df8f67070b262e71"} Oct 29 22:57:24.256250 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:24.256212 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" event={"ID":"9a4615b1-ff3d-4339-853a-f5d5247a5573","Type":"ContainerStarted","Data":"a12fb4555f10562286caac8eb77e90e5213b66abb1fe7466eb5002a166a6eea4"} Oct 29 22:57:24.306145 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:24.306095 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" podStartSLOduration=1.306080559 podStartE2EDuration="1.306080559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:24.303517663 +0000 UTC m=+890.955016184" watchObservedRunningTime="2026-10-29 22:57:24.306080559 +0000 UTC m=+890.957579088" Oct 29 22:57:24.615343 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:24.615252 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:25.616860 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:25.616827 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:57:25.617300 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:25.616879 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" podUID="9a4615b1-ff3d-4339-853a-f5d5247a5573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 29 22:57:26.615483 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:26.615400 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:57:26.615483 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:26.615452 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" podUID="9a4615b1-ff3d-4339-853a-f5d5247a5573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 29 22:57:27.615741 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:27.615708 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:57:27.616313 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:27.615769 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" podUID="9a4615b1-ff3d-4339-853a-f5d5247a5573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 29 22:57:28.274663 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.274624 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6"] Oct 29 22:57:28.279469 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.279453 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.302941 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.302898 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6"] Oct 29 22:57:28.318407 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.318379 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.318589 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.318430 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.318589 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.318529 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrqjh\" (UniqueName: \"kubernetes.io/projected/e0c58b99-cc75-42a7-8ba1-8734214ca340-kube-api-access-mrqjh\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.318589 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.318552 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.318784 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.318675 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.318784 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.318703 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.318784 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.318736 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.318940 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.318814 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0c58b99-cc75-42a7-8ba1-8734214ca340-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.318940 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.318844 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.420237 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.420177 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.420435 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.420273 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.420435 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.420293 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.420435 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.420318 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.420435 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.420347 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0c58b99-cc75-42a7-8ba1-8734214ca340-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.420435 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.420372 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.420706 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.420456 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.420706 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.420493 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.420706 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.420552 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrqjh\" (UniqueName: \"kubernetes.io/projected/e0c58b99-cc75-42a7-8ba1-8734214ca340-kube-api-access-mrqjh\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.421098 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.421044 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.437213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.421271 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.437213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.421271 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.437735 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.437705 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.452229 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.451663 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0c58b99-cc75-42a7-8ba1-8734214ca340-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.458620 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.458583 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.458780 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.458762 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.459530 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.458919 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrqjh\" (UniqueName: \"kubernetes.io/projected/e0c58b99-cc75-42a7-8ba1-8734214ca340-kube-api-access-mrqjh\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.461073 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.461050 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.596563 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.596477 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:28.615252 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.615219 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:57:28.615368 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.615287 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" podUID="9a4615b1-ff3d-4339-853a-f5d5247a5573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 29 22:57:28.764264 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.764214 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr"] Oct 29 22:57:28.770720 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.770690 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" Oct 29 22:57:28.831317 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.830523 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-79e6c631-serving-cert\"" Oct 29 22:57:28.860576 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.860482 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kplcr\" (UniqueName: \"kubernetes.io/projected/7723f51d-300b-486d-9ce1-e69a034e0939-kube-api-access-kplcr\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr\" (UID: \"7723f51d-300b-486d-9ce1-e69a034e0939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" Oct 29 22:57:28.860849 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.860825 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7723f51d-300b-486d-9ce1-e69a034e0939-metrics-certs\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr\" (UID: \"7723f51d-300b-486d-9ce1-e69a034e0939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" Oct 29 22:57:28.860991 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.860972 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7723f51d-300b-486d-9ce1-e69a034e0939-maas-api-tls\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr\" (UID: \"7723f51d-300b-486d-9ce1-e69a034e0939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" Oct 29 22:57:28.861370 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.861349 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7723f51d-300b-486d-9ce1-e69a034e0939-openshift-service-ca\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr\" (UID: \"7723f51d-300b-486d-9ce1-e69a034e0939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" Oct 29 22:57:28.884440 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.884351 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr"] Oct 29 22:57:28.963806 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.963659 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7723f51d-300b-486d-9ce1-e69a034e0939-openshift-service-ca\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr\" (UID: \"7723f51d-300b-486d-9ce1-e69a034e0939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" Oct 29 22:57:28.963806 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.963734 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kplcr\" (UniqueName: \"kubernetes.io/projected/7723f51d-300b-486d-9ce1-e69a034e0939-kube-api-access-kplcr\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr\" (UID: \"7723f51d-300b-486d-9ce1-e69a034e0939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" Oct 29 22:57:28.963806 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.963792 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7723f51d-300b-486d-9ce1-e69a034e0939-metrics-certs\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr\" (UID: \"7723f51d-300b-486d-9ce1-e69a034e0939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" Oct 29 22:57:28.963931 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.963820 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7723f51d-300b-486d-9ce1-e69a034e0939-maas-api-tls\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr\" (UID: \"7723f51d-300b-486d-9ce1-e69a034e0939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" Oct 29 22:57:28.964023 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:57:28.963996 5057 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-serving-cert: secret "maas-api-e2e-mt-79e6c631-serving-cert" not found Oct 29 22:57:28.964083 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:57:28.964068 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7723f51d-300b-486d-9ce1-e69a034e0939-maas-api-tls podName:7723f51d-300b-486d-9ce1-e69a034e0939 nodeName:}" failed. No retries permitted until 2026-10-29 22:57:29.464048601 +0000 UTC m=+896.115547115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7723f51d-300b-486d-9ce1-e69a034e0939-maas-api-tls") pod "maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" (UID: "7723f51d-300b-486d-9ce1-e69a034e0939") : secret "maas-api-e2e-mt-79e6c631-serving-cert" not found Oct 29 22:57:28.967252 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:57:28.966680 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c58b99_cc75_42a7_8ba1_8734214ca340.slice/crio-3fd7e1e52fa687b2b1d1192e48d8b3a4a2f85a9203921cc46330541515f4e312 WatchSource:0}: Error finding container 3fd7e1e52fa687b2b1d1192e48d8b3a4a2f85a9203921cc46330541515f4e312: Status 404 returned error can't find the container with id 3fd7e1e52fa687b2b1d1192e48d8b3a4a2f85a9203921cc46330541515f4e312 Oct 29 22:57:28.968425 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.968053 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7723f51d-300b-486d-9ce1-e69a034e0939-openshift-service-ca\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr\" (UID: \"7723f51d-300b-486d-9ce1-e69a034e0939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" Oct 29 22:57:28.973111 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.970336 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6"] Oct 29 22:57:28.973111 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.970487 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7723f51d-300b-486d-9ce1-e69a034e0939-metrics-certs\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr\" (UID: \"7723f51d-300b-486d-9ce1-e69a034e0939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" Oct 29 22:57:28.977967 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.977889 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:28.978131 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.978104 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:28.978272 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.978239 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:28.997206 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:28.996343 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kplcr\" (UniqueName: \"kubernetes.io/projected/7723f51d-300b-486d-9ce1-e69a034e0939-kube-api-access-kplcr\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr\" (UID: \"7723f51d-300b-486d-9ce1-e69a034e0939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" Oct 29 22:57:29.254730 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:29.253467 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2" Oct 29 22:57:29.319579 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:29.318382 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" event={"ID":"e0c58b99-cc75-42a7-8ba1-8734214ca340","Type":"ContainerStarted","Data":"2fb7a38252a53a5b475c35550ff5beb9cdbbd5fdb1707f81621892c0d3f0800b"} Oct 29 22:57:29.319579 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:29.318453 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" event={"ID":"e0c58b99-cc75-42a7-8ba1-8734214ca340","Type":"ContainerStarted","Data":"3fd7e1e52fa687b2b1d1192e48d8b3a4a2f85a9203921cc46330541515f4e312"} Oct 29 22:57:29.413849 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:29.412175 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" podStartSLOduration=1.41215485 podStartE2EDuration="1.41215485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:29.409463914 +0000 UTC m=+896.060962442" watchObservedRunningTime="2026-10-29 22:57:29.41215485 +0000 UTC m=+896.063653381" Oct 29 22:57:29.490214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:29.489736 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7723f51d-300b-486d-9ce1-e69a034e0939-maas-api-tls\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr\" (UID: \"7723f51d-300b-486d-9ce1-e69a034e0939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" Oct 29 22:57:29.493230 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:29.493027 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7723f51d-300b-486d-9ce1-e69a034e0939-maas-api-tls\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr\" (UID: \"7723f51d-300b-486d-9ce1-e69a034e0939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" Oct 29 22:57:29.596696 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:29.596546 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:29.672378 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:29.617779 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:57:29.679211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:29.673276 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" podUID="9a4615b1-ff3d-4339-853a-f5d5247a5573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 29 22:57:29.694534 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:29.694503 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" Oct 29 22:57:30.413709 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.406465 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr"] Oct 29 22:57:30.415123 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:57:30.414422 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7723f51d_300b_486d_9ce1_e69a034e0939.slice/crio-7d6e575fa0f81162dc6bd8a69fba8adf225f25eb861367be932d831349da8c0b WatchSource:0}: Error finding container 7d6e575fa0f81162dc6bd8a69fba8adf225f25eb861367be932d831349da8c0b: Status 404 returned error can't find the container with id 7d6e575fa0f81162dc6bd8a69fba8adf225f25eb861367be932d831349da8c0b Oct 29 22:57:30.459234 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.431937 5057 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:57:30.584211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.574441 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652"] Oct 29 22:57:30.620217 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.610304 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:57:30.620217 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.610368 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" podUID="e0c58b99-cc75-42a7-8ba1-8734214ca340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 29 22:57:30.675325 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.674336 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:57:30.675325 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.674439 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" podUID="9a4615b1-ff3d-4339-853a-f5d5247a5573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 29 22:57:30.732209 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.730880 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652"] Oct 29 22:57:30.750208 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.747315 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.812819 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.807289 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c8812a-4547-46fc-ad21-ab74038a0fae-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.813103 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.813076 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvc5g\" (UniqueName: \"kubernetes.io/projected/a5c8812a-4547-46fc-ad21-ab74038a0fae-kube-api-access-rvc5g\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.813297 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.813278 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.813424 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.813409 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.813646 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.813624 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.813841 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.813801 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.813999 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.813980 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.814132 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.814118 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.814279 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.814262 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.915503 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.915470 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.915698 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.915513 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.915698 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.915571 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.915698 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.915610 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.915698 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.915650 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.915698 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.915695 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.916011 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.915724 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.916011 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.915845 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c8812a-4547-46fc-ad21-ab74038a0fae-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.916011 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.915881 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.916011 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.915894 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvc5g\" (UniqueName: \"kubernetes.io/projected/a5c8812a-4547-46fc-ad21-ab74038a0fae-kube-api-access-rvc5g\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.916011 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.915940 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.916480 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.916458 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.916763 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.916728 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c8812a-4547-46fc-ad21-ab74038a0fae-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.916861 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.916801 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:30.918202 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.918162 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:31.004731 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:30.985441 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvc5g\" (UniqueName: \"kubernetes.io/projected/a5c8812a-4547-46fc-ad21-ab74038a0fae-kube-api-access-rvc5g\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:31.049478 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.020645 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:31.121179 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.121138 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:31.217582 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.194242 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68"] Oct 29 22:57:31.220697 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.220656 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68"] Oct 29 22:57:31.221042 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.221027 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.254207 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.226756 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.270415 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.266343 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.270591 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.270404 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.270672 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.270590 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.270672 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.270646 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjrhc\" (UniqueName: \"kubernetes.io/projected/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-kube-api-access-gjrhc\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.270792 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.270782 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.270849 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.270824 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.270906 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.270872 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.270906 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.270896 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.376254 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.375974 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.376254 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.376042 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.376254 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.376061 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjrhc\" (UniqueName: \"kubernetes.io/projected/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-kube-api-access-gjrhc\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.376254 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.376133 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.376254 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.376155 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.376254 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.376221 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.376254 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.376249 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.376981 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.376320 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.376981 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.376340 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.377098 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.376999 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.377872 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.377258 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.377872 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.377539 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.378054 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.377875 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.378165 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.378119 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.381346 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.381309 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.381592 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.381571 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.403208 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.402372 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.403208 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.403124 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjrhc\" (UniqueName: \"kubernetes.io/projected/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-kube-api-access-gjrhc\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.409371 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.409149 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:31.582092 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.573463 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" event={"ID":"7723f51d-300b-486d-9ce1-e69a034e0939","Type":"ContainerStarted","Data":"89ce811176c5fa41dab630fa9f169c0e69e2455df156a928086192041454890d"} Oct 29 22:57:31.582092 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.573524 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" event={"ID":"7723f51d-300b-486d-9ce1-e69a034e0939","Type":"ContainerStarted","Data":"7d6e575fa0f81162dc6bd8a69fba8adf225f25eb861367be932d831349da8c0b"} Oct 29 22:57:31.582092 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.573573 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" Oct 29 22:57:31.595646 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.595580 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:31.597352 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.597072 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:57:31.597557 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.597533 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" podUID="e0c58b99-cc75-42a7-8ba1-8734214ca340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 29 22:57:31.691956 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.691262 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" podStartSLOduration=2.8401027880000003 podStartE2EDuration="3.69124306s" totalImagesPullingTime="851.140272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:31.685629978 +0000 UTC m=+898.337128509" watchObservedRunningTime="2026-10-29 22:57:31.69124306 +0000 UTC m=+898.342741591" Oct 29 22:57:31.981225 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:31.976648 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652"] Oct 29 22:57:32.090870 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:57:32.090829 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c8812a_4547_46fc_ad21_ab74038a0fae.slice/crio-d58aa59af5efe7aef3bd071476a1abdacb0cf20223ac6e0e704659aab7a07cf2 WatchSource:0}: Error finding container d58aa59af5efe7aef3bd071476a1abdacb0cf20223ac6e0e704659aab7a07cf2: Status 404 returned error can't find the container with id d58aa59af5efe7aef3bd071476a1abdacb0cf20223ac6e0e704659aab7a07cf2 Oct 29 22:57:32.094362 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:32.094321 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:32.094454 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:32.094439 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:32.094527 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:32.094501 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:32.120159 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:32.120132 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:32.122118 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:32.122099 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:32.127069 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:32.127022 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 22:57:32.292079 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:32.285480 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:32.292079 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:32.285633 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:32.292079 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:32.285690 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:32.292079 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:32.291367 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68"] Oct 29 22:57:32.586887 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:32.586803 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" event={"ID":"51d78fc8-72e3-468e-a7d5-b3ecdeaece75","Type":"ContainerStarted","Data":"0e28d0bbf8ba9ed33195dd5ec647561118eec18c4fc31b859e1c483390b5b555"} Oct 29 22:57:32.586887 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:32.586857 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" event={"ID":"51d78fc8-72e3-468e-a7d5-b3ecdeaece75","Type":"ContainerStarted","Data":"0b7cf17088305928db707c5344e1d4b5948934d683d8ee80ea49caac042725d9"} Oct 29 22:57:32.590128 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:32.590099 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" event={"ID":"a5c8812a-4547-46fc-ad21-ab74038a0fae","Type":"ContainerStarted","Data":"2c6d619f9a8f0b35d19d25773e3f12ddd4a4fa2e80f12e49d749bf3ac4947af3"} Oct 29 22:57:32.590281 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:32.590141 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" event={"ID":"a5c8812a-4547-46fc-ad21-ab74038a0fae","Type":"ContainerStarted","Data":"d58aa59af5efe7aef3bd071476a1abdacb0cf20223ac6e0e704659aab7a07cf2"} Oct 29 22:57:32.596915 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:32.596893 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:32.597178 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:32.597145 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:57:32.597299 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:32.597239 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" podUID="e0c58b99-cc75-42a7-8ba1-8734214ca340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 29 22:57:32.656571 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:32.656124 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" podStartSLOduration=2.656104195 podStartE2EDuration="2.656104195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:32.641636392 +0000 UTC m=+899.293134923" watchObservedRunningTime="2026-10-29 22:57:32.656104195 +0000 UTC m=+899.307602725" Oct 29 22:57:32.697175 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:32.696613 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podStartSLOduration=2.696595835 podStartE2EDuration="2.696595835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:32.690943634 +0000 UTC m=+899.342442165" watchObservedRunningTime="2026-10-29 22:57:32.696595835 +0000 UTC m=+899.348094366" Oct 29 22:57:33.415477 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:33.415417 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:33.417792 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:33.417757 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:33.417963 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:33.417811 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:57:33.607118 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:33.606776 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:33.607118 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:33.606836 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:57:33.608211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:33.608072 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:57:33.608211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:33.608141 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" podUID="e0c58b99-cc75-42a7-8ba1-8734214ca340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 29 22:57:34.140213 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:57:34.121821 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:57:34.414248 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:34.414041 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:34.414584 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:34.414456 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:57:34.491180 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:34.491149 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 22:57:34.518616 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:34.518577 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 22:57:34.610209 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:34.596796 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:57:34.610209 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:34.596849 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" podUID="e0c58b99-cc75-42a7-8ba1-8734214ca340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 29 22:57:34.610209 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:34.597295 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:34.610209 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:34.597332 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:57:35.416333 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:35.416293 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:35.416543 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:35.416352 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:57:35.611092 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:35.611044 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:35.624455 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:35.624292 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:57:35.659227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:35.643602 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:57:35.659227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:35.643660 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" podUID="e0c58b99-cc75-42a7-8ba1-8734214ca340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 29 22:57:36.145210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:36.143855 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:57:36.413928 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:36.413846 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:36.414166 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:36.414137 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:57:36.627345 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:36.627309 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:57:36.642321 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:36.642281 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" podUID="e0c58b99-cc75-42a7-8ba1-8734214ca340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 29 22:57:36.643180 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:36.643158 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:36.643380 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:36.643358 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:57:37.435438 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:37.435376 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:37.455765 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:37.435705 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:57:37.601213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:37.600203 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:37.601213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:37.600375 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:57:37.629207 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:37.616438 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:57:37.629207 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:37.616495 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" podUID="e0c58b99-cc75-42a7-8ba1-8734214ca340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 29 22:57:38.429739 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:38.429707 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:38.445257 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:38.429986 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:57:38.649318 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:38.649273 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 22:57:38.696269 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:38.668362 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:38.696269 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:38.668423 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:57:38.696269 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:38.668520 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:57:38.696269 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:38.668546 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" podUID="e0c58b99-cc75-42a7-8ba1-8734214ca340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 29 22:57:38.832330 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:38.832277 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh"] Oct 29 22:57:39.217030 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:38.874439 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:39.217532 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:38.874495 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:39.236294 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:38.987259 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:39.380846 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:39.380792 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" Oct 29 22:57:39.381139 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:39.381099 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh"] Oct 29 22:57:39.386336 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:39.386311 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:39.453838 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:39.453787 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:39.454257 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:39.454220 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:57:39.649468 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:39.625205 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:39.649468 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:39.625277 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:57:39.669879 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:39.655892 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:39.669879 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:39.655964 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:39.669879 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:39.656171 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:39.669879 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:39.656398 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:39.669879 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:39.656449 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:39.675235 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:39.674661 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:39.935280 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:39.910723 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:39.935819 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:39.935775 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:40.057624 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.057570 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:40.057980 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.057959 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:40.058202 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.058158 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:40.085613 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.085581 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:40.158550 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.158518 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:40.194366 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:39.937092 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:40.195042 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.195012 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:40.198057 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.198032 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:40.240841 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.240790 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:40.351245 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.341711 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" podUID="f7feb618-94bf-4988-abca-4dc4fbd699c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:40.351416 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.351254 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 22:57:40.424228 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.404343 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:40.491718 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.425983 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 22:57:40.491718 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.426350 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 22:57:40.491718 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.427668 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:40.491718 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.427711 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:57:40.494698 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.493716 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:40.622214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.597733 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:40.622510 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.622475 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:57:40.652212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.652018 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33c7a18b-c654-4401-88da-bd2875dd61c9-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:40.652212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.652129 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:40.652807 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.652287 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng9wq\" (UniqueName: \"kubernetes.io/projected/33c7a18b-c654-4401-88da-bd2875dd61c9-kube-api-access-ng9wq\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:40.652807 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.652357 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:40.734212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.730267 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 22:57:40.765366 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.765250 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng9wq\" (UniqueName: \"kubernetes.io/projected/33c7a18b-c654-4401-88da-bd2875dd61c9-kube-api-access-ng9wq\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:40.765801 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.765769 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:40.774347 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.774318 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33c7a18b-c654-4401-88da-bd2875dd61c9-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:40.807637 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.807586 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33c7a18b-c654-4401-88da-bd2875dd61c9-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:40.869238 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:57:40.856220 5057 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.366s" Oct 29 22:57:40.869238 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.856344 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:57:40.869238 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.856413 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:40.869238 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.856473 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:57:40.869238 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.856621 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:57:40.954756 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.954716 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:40.955010 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:40.954980 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" podUID="ed5d8696-66e9-4e1d-9419-2597fef50ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:41.256214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:41.241028 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:41.289280 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:41.241433 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:41.401653 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:41.401604 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:41.429026 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:41.428988 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng9wq\" (UniqueName: \"kubernetes.io/projected/33c7a18b-c654-4401-88da-bd2875dd61c9-kube-api-access-ng9wq\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:41.451214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:41.450014 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:41.461886 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:41.461840 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:41.462112 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:41.461908 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" podUID="e0c58b99-cc75-42a7-8ba1-8734214ca340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:41.483761 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:41.483711 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:41.486962 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:41.486923 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:41.487108 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:41.486980 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:57:41.494233 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:41.488240 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:41.599314 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:41.598329 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:41.599314 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:41.598393 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:57:41.795165 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:41.776545 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:42.446208 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:42.428493 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:42.446208 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:42.428603 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:57:42.449249 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:42.448743 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:42.480213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:42.455110 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 22:57:42.485366 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:42.485268 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:42.533212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:42.525302 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:42.533212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:42.525386 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" podUID="e0c58b99-cc75-42a7-8ba1-8734214ca340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:42.624068 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:42.624029 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:42.624406 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:42.624376 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:57:42.800269 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:42.792296 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 22:57:43.197264 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:43.178907 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-nlz8l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:43.197264 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:43.178985 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" podUID="196b11f8-6a98-45c1-800b-8de400a902bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:43.412244 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:43.411622 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:43.412244 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:43.411698 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:57:43.466696 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:43.450278 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 22:57:43.531211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:43.522283 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 22:57:43.536413 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:43.536374 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:43.590987 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:43.586289 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:43.590987 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:43.586363 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" podUID="e0c58b99-cc75-42a7-8ba1-8734214ca340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:43.649281 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:43.631139 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:43.649281 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:43.631216 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:57:43.657736 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:43.653282 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:43.657736 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:43.653346 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" podUID="4193106b-ad8c-4e1c-bd90-3a244651d979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:43.805299 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:43.801649 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 22:57:44.098266 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:44.098083 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:44.123212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:44.098450 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:44.123212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:44.110178 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 22:57:44.484212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:44.422105 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:44.639580 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:44.639522 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:57:44.677267 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:44.519990 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:44.716413 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:44.536843 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:44.716413 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:44.593167 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:44.716413 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:44.524812 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:44.716413 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:44.614956 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:44.716413 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:44.682145 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:57:44.945730 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:44.830246 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:44.945730 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:44.922288 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:44.972617 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:44.972436 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:45.087211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:45.053775 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:45.245717 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:45.245657 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh"] Oct 29 22:57:45.245892 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:45.245809 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:45.521270 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:45.519181 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:45.521270 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:45.519258 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:57:45.630237 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:45.606348 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:45.630237 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:45.606431 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:57:45.695725 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:45.695677 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:45.699281 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:45.695977 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 22:57:45.699817 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:45.696028 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:45.970601 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:45.968881 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" podUID="907c36dd-72ba-42b8-86f1-9ff84400b4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 29 22:57:45.970601 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:45.969310 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" podUID="5f7503b3-a2b7-4795-b78a-9f18d00b5f2f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:46.031333 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:46.031270 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" podUID="bce69276-30ac-47e6-99df-81608424b3dd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:46.064042 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:46.062294 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:46.106275 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:46.092399 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:46.220724 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:46.220680 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" event={"ID":"33c7a18b-c654-4401-88da-bd2875dd61c9","Type":"ContainerStarted","Data":"6bc1cda4b7bb31200ddd29972ef2d8ba9397e50045bbbb36f62b768ea194680d"} Oct 29 22:57:46.413411 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:46.413315 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:46.420249 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:46.420218 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:46.420508 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:46.420467 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:57:46.596365 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:46.596327 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:46.745224 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:46.718357 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:46.745224 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:46.718741 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:46.745224 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:46.720172 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:46.746745 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:46.746714 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:46.746979 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:46.746946 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:57:46.758362 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:46.758325 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" event={"ID":"33c7a18b-c654-4401-88da-bd2875dd61c9","Type":"ContainerStarted","Data":"05f8ed87ac01b9b4f37447b05e30fe91682bbb60886b5c6da81cb2513265c38f"} Oct 29 22:57:46.977026 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:46.976987 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:57:46.979532 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:46.979496 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:57:46.988598 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:46.988575 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:57:47.084263 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:47.081252 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" podStartSLOduration=9.039180016 podStartE2EDuration="9.039180016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:46.84066662 +0000 UTC m=+913.492165151" watchObservedRunningTime="2026-10-29 22:57:47.039180016 +0000 UTC m=+913.690678550" Oct 29 22:57:47.453211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:47.449507 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:47.467164 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:47.467124 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:57:47.570020 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:47.568341 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:57:47.570020 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:47.568404 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" podUID="33c7a18b-c654-4401-88da-bd2875dd61c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 29 22:57:47.628578 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:47.628546 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:47.628814 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:47.628785 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:57:47.756264 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:47.731082 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:47.886244 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:47.825907 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4"] Oct 29 22:57:48.025387 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.002340 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:48.025387 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.016135 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4"] Oct 29 22:57:48.025387 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.016207 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:48.025387 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.016317 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.188391 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.184631 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c79a4e-openshift-default-dockercfg-5qpsg\"" Oct 29 22:57:48.212244 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.209393 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-podinfo\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.212244 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.209477 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istiod-ca-cert\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.212244 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.209546 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv8kf\" (UniqueName: \"kubernetes.io/projected/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-kube-api-access-gv8kf\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.212244 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.209580 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-envoy\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.212244 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.209623 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-token\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.212244 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.209684 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-workload-socket\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.212244 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.209753 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-workload-certs\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.212244 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.209893 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-credential-socket\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.212244 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.209949 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-data\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.293827 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.293731 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:57:48.322755 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.322702 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv8kf\" (UniqueName: \"kubernetes.io/projected/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-kube-api-access-gv8kf\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.541338 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.421854 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:48.541628 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.541592 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:57:48.541759 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.491764 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:48.547025 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.491111 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 22:57:48.548218 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.545443 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv8kf\" (UniqueName: \"kubernetes.io/projected/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-kube-api-access-gv8kf\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.548218 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.501012 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:57:48.548218 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.547525 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" podUID="33c7a18b-c654-4401-88da-bd2875dd61c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 29 22:57:48.548218 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.546633 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-envoy\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.548218 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.547622 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-token\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.548218 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.547715 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-workload-socket\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.548218 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.547810 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-workload-certs\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.548218 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.548017 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-credential-socket\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.548218 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.548103 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-data\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.548218 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.548147 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-podinfo\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.552211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.548959 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istiod-ca-cert\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.552211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.549755 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istiod-ca-cert\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.552211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.550853 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-workload-socket\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.552211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.551288 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-workload-certs\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.552211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.551882 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-credential-socket\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.552797 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.552775 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-data\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:48.628687 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.628653 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:48.628981 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.628949 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 29 22:57:48.785212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:48.779356 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-token\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:49.099212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:49.076738 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-envoy\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:49.199215 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:49.189044 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-podinfo\") pod \"e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:49.484268 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:49.441653 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:49.501203 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:49.501158 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:57:49.501474 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:49.501443 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" podUID="33c7a18b-c654-4401-88da-bd2875dd61c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 29 22:57:50.434224 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:50.429623 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:50.504303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:50.504262 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:50.505134 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:50.505098 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:50.506685 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:50.506661 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:57:50.506789 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:50.506700 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" podUID="33c7a18b-c654-4401-88da-bd2875dd61c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 29 22:57:50.649212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:50.648787 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:50.649212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:50.648860 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:50.783806 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:50.783671 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:50.785419 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:50.785399 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 22:57:50.997501 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:50.997449 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:50.997638 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:50.997550 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:50.997638 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:50.997595 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:57:51.000290 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:50.998646 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4"] Oct 29 22:57:51.503292 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:51.501600 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:57:51.503292 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:51.501661 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" podUID="33c7a18b-c654-4401-88da-bd2875dd61c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 29 22:57:51.552717 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:51.552318 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:51.552717 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:51.552382 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:51.707338 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:51.707302 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" event={"ID":"e3cd3c9e-4e7d-4395-81da-5f43c01467ee","Type":"ContainerStarted","Data":"0dee6d22d5168608675727d1786f2d4099b04cb69acee06543ee19eecd266452"} Oct 29 22:57:51.707623 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:51.707575 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" event={"ID":"e3cd3c9e-4e7d-4395-81da-5f43c01467ee","Type":"ContainerStarted","Data":"74dfea150b5cdb507c18addc5d661c502d6e62d0fa83b97e5684e19f92b27b57"} Oct 29 22:57:52.467929 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:52.467845 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:52.476825 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:52.476510 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:52.525255 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:52.517907 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:57:52.525255 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:52.517984 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" podUID="33c7a18b-c654-4401-88da-bd2875dd61c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 29 22:57:52.589208 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:52.585792 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" podStartSLOduration=5.585770293 podStartE2EDuration="5.585770293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:51.773504541 +0000 UTC m=+918.425003071" watchObservedRunningTime="2026-10-29 22:57:52.585770293 +0000 UTC m=+919.237268831" Oct 29 22:57:52.644011 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:52.639313 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:52.644011 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:52.639391 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:52.665628 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:52.665583 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:52.679524 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:52.679482 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 22:57:52.715614 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:52.715589 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:52.716600 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:52.716578 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:57:53.496325 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:53.496292 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:57:53.496325 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:53.496349 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" podUID="33c7a18b-c654-4401-88da-bd2875dd61c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 29 22:57:54.496986 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:54.496951 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:57:54.497379 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:54.497002 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" podUID="33c7a18b-c654-4401-88da-bd2875dd61c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 29 22:57:55.496683 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:55.496646 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:57:55.496862 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:55.496712 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" podUID="33c7a18b-c654-4401-88da-bd2875dd61c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 29 22:57:56.208452 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.208408 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk"] Oct 29 22:57:56.214141 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.214115 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk" Oct 29 22:57:56.234441 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.234410 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk"] Oct 29 22:57:56.244366 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.244330 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c79a4e-dockercfg-v6wft\"" Oct 29 22:57:56.244366 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.244353 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c79a4e\"" Oct 29 22:57:56.287951 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.287917 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9"] Oct 29 22:57:56.292387 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.292363 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9" Oct 29 22:57:56.298109 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.298077 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffvkb\" (UniqueName: \"kubernetes.io/projected/7b24c9e7-cc9d-4f56-a440-aaec87ee199b-kube-api-access-ffvkb\") pod \"payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk\" (UID: \"7b24c9e7-cc9d-4f56-a440-aaec87ee199b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk" Oct 29 22:57:56.298241 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.298168 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b24c9e7-cc9d-4f56-a440-aaec87ee199b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk\" (UID: \"7b24c9e7-cc9d-4f56-a440-aaec87ee199b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk" Oct 29 22:57:56.323487 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.319525 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9"] Oct 29 22:57:56.634307 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.627164 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffvkb\" (UniqueName: \"kubernetes.io/projected/7b24c9e7-cc9d-4f56-a440-aaec87ee199b-kube-api-access-ffvkb\") pod \"payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk\" (UID: \"7b24c9e7-cc9d-4f56-a440-aaec87ee199b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk" Oct 29 22:57:56.634637 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.634609 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b24c9e7-cc9d-4f56-a440-aaec87ee199b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk\" (UID: \"7b24c9e7-cc9d-4f56-a440-aaec87ee199b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk" Oct 29 22:57:56.636164 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.635926 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b24c9e7-cc9d-4f56-a440-aaec87ee199b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk\" (UID: \"7b24c9e7-cc9d-4f56-a440-aaec87ee199b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk" Oct 29 22:57:56.646958 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.638716 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:56.666965 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.666929 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffvkb\" (UniqueName: \"kubernetes.io/projected/7b24c9e7-cc9d-4f56-a440-aaec87ee199b-kube-api-access-ffvkb\") pod \"payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk\" (UID: \"7b24c9e7-cc9d-4f56-a440-aaec87ee199b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk" Oct 29 22:57:56.670782 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.670756 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:56.736427 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.736390 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbsfv\" (UniqueName: \"kubernetes.io/projected/998b1e6e-e9fc-493b-aca5-2cf89f43cc04-kube-api-access-gbsfv\") pod \"payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9\" (UID: \"998b1e6e-e9fc-493b-aca5-2cf89f43cc04\") " pod="openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9" Oct 29 22:57:56.736689 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.736655 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998b1e6e-e9fc-493b-aca5-2cf89f43cc04-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9\" (UID: \"998b1e6e-e9fc-493b-aca5-2cf89f43cc04\") " pod="openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9" Oct 29 22:57:56.771446 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.771422 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 22:57:56.828123 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.828091 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk" Oct 29 22:57:56.838738 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.838705 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998b1e6e-e9fc-493b-aca5-2cf89f43cc04-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9\" (UID: \"998b1e6e-e9fc-493b-aca5-2cf89f43cc04\") " pod="openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9" Oct 29 22:57:56.841091 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.841061 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbsfv\" (UniqueName: \"kubernetes.io/projected/998b1e6e-e9fc-493b-aca5-2cf89f43cc04-kube-api-access-gbsfv\") pod \"payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9\" (UID: \"998b1e6e-e9fc-493b-aca5-2cf89f43cc04\") " pod="openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9" Oct 29 22:57:56.841825 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.841792 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998b1e6e-e9fc-493b-aca5-2cf89f43cc04-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9\" (UID: \"998b1e6e-e9fc-493b-aca5-2cf89f43cc04\") " pod="openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9" Oct 29 22:57:56.886305 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.886176 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbsfv\" (UniqueName: \"kubernetes.io/projected/998b1e6e-e9fc-493b-aca5-2cf89f43cc04-kube-api-access-gbsfv\") pod \"payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9\" (UID: \"998b1e6e-e9fc-493b-aca5-2cf89f43cc04\") " pod="openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9" Oct 29 22:57:56.922893 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:56.922622 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9" Oct 29 22:57:57.158494 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:57.158465 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk"] Oct 29 22:57:57.159960 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:57:57.159933 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b24c9e7_cc9d_4f56_a440_aaec87ee199b.slice/crio-4ca174e1582c0e310b4c830dc8019a10f44c11d0d63b28b91a4f92441d2acee0 WatchSource:0}: Error finding container 4ca174e1582c0e310b4c830dc8019a10f44c11d0d63b28b91a4f92441d2acee0: Status 404 returned error can't find the container with id 4ca174e1582c0e310b4c830dc8019a10f44c11d0d63b28b91a4f92441d2acee0 Oct 29 22:57:57.259642 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:57.259610 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9"] Oct 29 22:57:57.783206 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:57.781574 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9" event={"ID":"998b1e6e-e9fc-493b-aca5-2cf89f43cc04","Type":"ContainerStarted","Data":"9ca5bb60d789b5c89f9c856897cc5fccb615ee21def3a3745f409bbf16e0d48e"} Oct 29 22:57:57.783206 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:57.781638 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9" event={"ID":"998b1e6e-e9fc-493b-aca5-2cf89f43cc04","Type":"ContainerStarted","Data":"006005ac9f155739e1b69c7a1f12baff4ecdce67710c03857b1169ac232558e9"} Oct 29 22:57:57.785254 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:57.785220 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk" event={"ID":"7b24c9e7-cc9d-4f56-a440-aaec87ee199b","Type":"ContainerStarted","Data":"98b67451818b26d1b75f18b1fa0d5a92a02497a7676a5c6a355b4a1ec341737f"} Oct 29 22:57:57.785571 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:57.785276 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk" event={"ID":"7b24c9e7-cc9d-4f56-a440-aaec87ee199b","Type":"ContainerStarted","Data":"4ca174e1582c0e310b4c830dc8019a10f44c11d0d63b28b91a4f92441d2acee0"} Oct 29 22:57:57.785571 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:57.785330 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk" Oct 29 22:57:57.913276 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:57.912435 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9" podStartSLOduration=1.9124147740000002 podStartE2EDuration="1.912414774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:57.853542516 +0000 UTC m=+924.505041052" watchObservedRunningTime="2026-10-29 22:57:57.912414774 +0000 UTC m=+924.563913306" Oct 29 22:57:57.917206 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:57.916824 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk" podStartSLOduration=1.916806137 podStartE2EDuration="1.916806137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:57.910587465 +0000 UTC m=+924.562085995" watchObservedRunningTime="2026-10-29 22:57:57.916806137 +0000 UTC m=+924.568304670" Oct 29 22:57:58.794735 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:57:58.794699 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9" Oct 29 22:58:03.914116 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:58:03.914065 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:58:04.140601 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.140563 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c"] Oct 29 22:58:04.145876 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.145854 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.170489 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.170416 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c"] Oct 29 22:58:04.228024 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.227974 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.228218 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.228108 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.228218 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.228159 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.228648 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.228620 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.228758 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.228706 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.228815 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.228760 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.229065 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.229039 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.229143 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.229085 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njzrc\" (UniqueName: \"kubernetes.io/projected/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-kube-api-access-njzrc\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.229243 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.229173 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.330649 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.330612 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.330825 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.330708 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.330825 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.330766 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.330825 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.330812 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.330987 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.330848 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njzrc\" (UniqueName: \"kubernetes.io/projected/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-kube-api-access-njzrc\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.330987 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.330942 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.330987 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.330969 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.331152 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.331041 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.331152 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.331088 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.331152 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.331100 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.331363 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.331177 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.331423 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.331407 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.331578 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.331557 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.331647 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.331623 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.333496 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.333469 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.333803 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.333781 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.356525 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.356501 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njzrc\" (UniqueName: \"kubernetes.io/projected/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-kube-api-access-njzrc\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.356674 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.356629 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-xbn5c\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.463369 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.463334 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:04.787351 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:58:04.787320 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb05fe4_ceb9_4a1c_8042_2a2b656eaa76.slice/crio-21f9f6d327a9fa53c70df54bbe81e81e56c88122c49ded4a5b0d9c7f9ac170a2 WatchSource:0}: Error finding container 21f9f6d327a9fa53c70df54bbe81e81e56c88122c49ded4a5b0d9c7f9ac170a2: Status 404 returned error can't find the container with id 21f9f6d327a9fa53c70df54bbe81e81e56c88122c49ded4a5b0d9c7f9ac170a2 Oct 29 22:58:04.787490 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.787470 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c"] Oct 29 22:58:04.790153 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.790118 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:58:04.790282 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.790211 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:58:04.790282 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.790243 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:58:04.861158 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:04.861116 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" event={"ID":"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76","Type":"ContainerStarted","Data":"21f9f6d327a9fa53c70df54bbe81e81e56c88122c49ded4a5b0d9c7f9ac170a2"} Oct 29 22:58:05.882989 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:05.882953 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" event={"ID":"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76","Type":"ContainerStarted","Data":"e4d59e053cf13523c9948891f7c5f4815f293a943d59f20bf4c93006d69e95a4"} Oct 29 22:58:05.960670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:05.960605 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" podStartSLOduration=1.960587799 podStartE2EDuration="1.960587799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:05.94538861 +0000 UTC m=+932.596887141" watchObservedRunningTime="2026-10-29 22:58:05.960587799 +0000 UTC m=+932.612086328" Oct 29 22:58:06.464515 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:06.464417 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:06.465895 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:06.465863 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-xbn5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 29 22:58:06.466065 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:06.465934 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" podUID="beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 29 22:58:07.038212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.037606 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9"] Oct 29 22:58:07.050218 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.048369 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.066515 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.066486 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9"] Oct 29 22:58:07.141212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.121656 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.141629 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.141595 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmthq\" (UniqueName: \"kubernetes.io/projected/84648cac-9c75-47a4-a3d6-df79e46b5fd9-kube-api-access-mmthq\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.141844 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.141823 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.142010 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.141995 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.149207 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.142257 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.159211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.149608 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.159476 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.159428 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.159575 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.159515 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.159575 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.159550 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.261227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.261169 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.261227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.261234 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.261480 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.261272 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.261480 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.261294 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.261480 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.261392 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.261480 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.261440 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmthq\" (UniqueName: \"kubernetes.io/projected/84648cac-9c75-47a4-a3d6-df79e46b5fd9-kube-api-access-mmthq\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.261686 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.261488 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.261686 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.261518 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.261686 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.261603 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.261686 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.261621 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.261827 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.261773 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.262160 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.262021 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.262333 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.262308 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.262393 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.262331 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.264525 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.264501 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.264878 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.264857 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.286889 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.286865 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.287022 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.286934 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmthq\" (UniqueName: \"kubernetes.io/projected/84648cac-9c75-47a4-a3d6-df79e46b5fd9-kube-api-access-mmthq\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.373359 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.373275 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:07.464498 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.464464 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-xbn5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 29 22:58:07.464619 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.464535 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" podUID="beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 29 22:58:07.740321 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.738284 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9"] Oct 29 22:58:07.758426 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.758390 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:58:07.758557 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.758459 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:58:07.758557 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.758488 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:58:07.911998 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.911968 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" event={"ID":"84648cac-9c75-47a4-a3d6-df79e46b5fd9","Type":"ContainerStarted","Data":"5ddb2149716a2f78db3992a65ab20f55e0e78c90ed3af04eefa7b7d6f6c9c388"} Oct 29 22:58:07.912218 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.912203 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" event={"ID":"84648cac-9c75-47a4-a3d6-df79e46b5fd9","Type":"ContainerStarted","Data":"796c8079892b6970579e9cb36f07a3b3dab7b5a2fb19ee1670d12d65183acd58"} Oct 29 22:58:07.967610 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:07.967560 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" podStartSLOduration=1.967544094 podStartE2EDuration="1.967544094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:07.964435121 +0000 UTC m=+934.615933652" watchObservedRunningTime="2026-10-29 22:58:07.967544094 +0000 UTC m=+934.619042624" Oct 29 22:58:08.373480 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:08.373443 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:08.374740 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:08.374713 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:58:08.374834 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:08.374762 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" podUID="84648cac-9c75-47a4-a3d6-df79e46b5fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:58:08.465382 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:08.465343 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-xbn5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 29 22:58:08.465565 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:08.465401 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" podUID="beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 29 22:58:08.808681 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:08.808650 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk" Oct 29 22:58:09.375703 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:09.375666 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:58:09.376361 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:09.376318 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" podUID="84648cac-9c75-47a4-a3d6-df79e46b5fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:58:09.466643 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:09.466608 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-xbn5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 29 22:58:09.466934 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:09.466907 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" podUID="beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 29 22:58:09.805367 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:09.805335 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9" Oct 29 22:58:10.092753 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.092659 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8"] Oct 29 22:58:10.098551 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.098520 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.118974 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.118841 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8"] Oct 29 22:58:10.202544 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.202510 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51473c56-3874-4c1b-af5b-0601a57b1d8f-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.202708 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.202560 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.202708 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.202601 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-545dw\" (UniqueName: \"kubernetes.io/projected/51473c56-3874-4c1b-af5b-0601a57b1d8f-kube-api-access-545dw\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.202708 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.202668 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.202708 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.202704 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.202840 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.202833 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.202988 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.202976 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.203033 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.203019 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.203098 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.203084 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.307214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.304762 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.307214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.304911 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.307214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.304947 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.307214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.304978 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.307214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.305044 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51473c56-3874-4c1b-af5b-0601a57b1d8f-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.307214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.305071 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.307214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.305118 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-545dw\" (UniqueName: \"kubernetes.io/projected/51473c56-3874-4c1b-af5b-0601a57b1d8f-kube-api-access-545dw\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.307214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.305210 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.307214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.305237 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.307799 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.307560 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.315214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.311352 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.315214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.311398 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51473c56-3874-4c1b-af5b-0601a57b1d8f-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.315214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.311796 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.315214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.312041 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.320834 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.320810 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.321038 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.321015 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.384279 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.381280 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.384279 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.381434 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:58:10.384279 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.381491 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" podUID="84648cac-9c75-47a4-a3d6-df79e46b5fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:58:10.399251 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.391590 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-545dw\" (UniqueName: \"kubernetes.io/projected/51473c56-3874-4c1b-af5b-0601a57b1d8f-kube-api-access-545dw\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.472253 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.472217 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-xbn5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 29 22:58:10.472516 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.472489 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" podUID="beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 29 22:58:10.514065 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.514036 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:10.589219 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.588074 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s"] Oct 29 22:58:10.594649 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.594625 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" Oct 29 22:58:10.610237 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.606410 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s"] Oct 29 22:58:10.610237 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.609598 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-openshift-service-ca\") pod \"maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s\" (UID: \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" Oct 29 22:58:10.610237 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.609716 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-metrics-certs\") pod \"maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s\" (UID: \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" Oct 29 22:58:10.610237 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.609781 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-maas-api-tls\") pod \"maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s\" (UID: \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" Oct 29 22:58:10.610237 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.609890 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5h9x\" (UniqueName: \"kubernetes.io/projected/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-kube-api-access-c5h9x\") pod \"maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s\" (UID: \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" Oct 29 22:58:10.611156 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.611131 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c79a4e-serving-cert\"" Oct 29 22:58:10.711304 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.711006 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-metrics-certs\") pod \"maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s\" (UID: \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" Oct 29 22:58:10.711583 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.711560 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-maas-api-tls\") pod \"maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s\" (UID: \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" Oct 29 22:58:10.712008 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.711968 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5h9x\" (UniqueName: \"kubernetes.io/projected/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-kube-api-access-c5h9x\") pod \"maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s\" (UID: \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" Oct 29 22:58:10.712255 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.712225 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-openshift-service-ca\") pod \"maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s\" (UID: \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" Oct 29 22:58:10.712761 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:58:10.711758 5057 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-serving-cert: secret "maas-api-e2e-api-a-c79a4e-serving-cert" not found Oct 29 22:58:10.712885 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:58:10.712868 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-maas-api-tls podName:4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7 nodeName:}" failed. No retries permitted until 2026-10-29 22:58:11.212843163 +0000 UTC m=+937.864341674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-maas-api-tls") pod "maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" (UID: "4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7") : secret "maas-api-e2e-api-a-c79a4e-serving-cert" not found Oct 29 22:58:10.713423 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.713402 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-openshift-service-ca\") pod \"maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s\" (UID: \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" Oct 29 22:58:10.797697 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:10.785227 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-metrics-certs\") pod \"maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s\" (UID: \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" Oct 29 22:58:11.060157 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:11.035850 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5h9x\" (UniqueName: \"kubernetes.io/projected/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-kube-api-access-c5h9x\") pod \"maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s\" (UID: \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" Oct 29 22:58:11.267221 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:11.265229 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-maas-api-tls\") pod \"maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s\" (UID: \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" Oct 29 22:58:11.267221 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:58:11.265602 5057 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-serving-cert: secret "maas-api-e2e-api-a-c79a4e-serving-cert" not found Oct 29 22:58:11.267221 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:58:11.265682 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-maas-api-tls podName:4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7 nodeName:}" failed. No retries permitted until 2026-10-29 22:58:12.265662456 +0000 UTC m=+938.917160985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-maas-api-tls") pod "maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" (UID: "4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7") : secret "maas-api-e2e-api-a-c79a4e-serving-cert" not found Oct 29 22:58:11.272777 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:58:11.271942 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51473c56_3874_4c1b_af5b_0601a57b1d8f.slice/crio-20b7a176fcfd4502463c8b0cb87ccc393b224f681444e221127cd968425299f6 WatchSource:0}: Error finding container 20b7a176fcfd4502463c8b0cb87ccc393b224f681444e221127cd968425299f6: Status 404 returned error can't find the container with id 20b7a176fcfd4502463c8b0cb87ccc393b224f681444e221127cd968425299f6 Oct 29 22:58:11.273631 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:11.273512 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8"] Oct 29 22:58:11.275497 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:11.275459 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:58:11.275577 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:11.275547 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:58:11.275616 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:11.275597 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:58:11.375434 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:11.375355 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:58:11.375434 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:11.375419 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" podUID="84648cac-9c75-47a4-a3d6-df79e46b5fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:58:11.464929 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:11.464795 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-xbn5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 29 22:58:11.464929 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:11.464858 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" podUID="beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 29 22:58:12.261212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:12.254050 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" event={"ID":"51473c56-3874-4c1b-af5b-0601a57b1d8f","Type":"ContainerStarted","Data":"e1701cd0fa6ef8f147d7dbf57b93387a4aff3aefb4f9db27af8605dc0c667788"} Oct 29 22:58:12.261212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:12.254122 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" event={"ID":"51473c56-3874-4c1b-af5b-0601a57b1d8f","Type":"ContainerStarted","Data":"20b7a176fcfd4502463c8b0cb87ccc393b224f681444e221127cd968425299f6"} Oct 29 22:58:12.351212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:12.330452 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-maas-api-tls\") pod \"maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s\" (UID: \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" Oct 29 22:58:12.385749 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:12.385682 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" podStartSLOduration=2.38565583 podStartE2EDuration="2.38565583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:12.355627367 +0000 UTC m=+939.007125898" watchObservedRunningTime="2026-10-29 22:58:12.38565583 +0000 UTC m=+939.037154361" Oct 29 22:58:12.386722 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:12.386670 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:58:12.386864 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:12.386741 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" podUID="84648cac-9c75-47a4-a3d6-df79e46b5fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:58:12.404227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:12.396943 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-maas-api-tls\") pod \"maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s\" (UID: \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" Oct 29 22:58:12.464596 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:12.464559 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-xbn5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 29 22:58:12.464836 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:12.464804 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" podUID="beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 29 22:58:12.515337 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:12.515253 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:12.517220 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:12.516494 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:58:12.517220 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:12.516557 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" podUID="51473c56-3874-4c1b-af5b-0601a57b1d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:58:12.613708 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:12.613667 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" Oct 29 22:58:13.333211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:13.326381 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s"] Oct 29 22:58:13.377212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:13.376287 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:58:13.377212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:13.376346 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" podUID="84648cac-9c75-47a4-a3d6-df79e46b5fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:58:13.497227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:13.492507 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-xbn5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 29 22:58:13.497227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:13.492562 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" podUID="beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 29 22:58:13.949212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:13.930335 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:58:13.949212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:13.930403 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" podUID="51473c56-3874-4c1b-af5b-0601a57b1d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:58:14.392233 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:14.392181 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:58:14.392536 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:14.392501 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" podUID="84648cac-9c75-47a4-a3d6-df79e46b5fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:58:14.425371 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:14.425242 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:14.470080 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:14.470031 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 22:58:14.528516 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:14.509483 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-xbn5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 29 22:58:14.528516 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:14.509560 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" podUID="beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 29 22:58:14.538207 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:14.531579 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:58:14.538207 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:14.531646 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" podUID="51473c56-3874-4c1b-af5b-0601a57b1d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:58:14.673670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:14.673624 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" event={"ID":"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7","Type":"ContainerStarted","Data":"cbb24be29097bba9e85b4f5837fd358acff89d373ca5557a3a0917fa35e3c2c1"} Oct 29 22:58:14.897253 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:14.896603 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:14.913213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:14.908537 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:15.224212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:15.215647 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:15.378653 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:15.378614 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:58:15.378921 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:15.378891 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" podUID="84648cac-9c75-47a4-a3d6-df79e46b5fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:58:15.430274 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:15.430171 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 22:58:15.431720 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:15.431696 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:58:15.478514 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:15.478355 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 22:58:15.730091 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:15.729992 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 22:58:15.878231 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:15.747068 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:58:15.878231 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:15.876727 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" podUID="51473c56-3874-4c1b-af5b-0601a57b1d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:58:15.909488 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:15.909438 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:15.924930 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:15.924885 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:16.480601 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:16.480563 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:58:16.481214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:16.481162 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" podUID="84648cac-9c75-47a4-a3d6-df79e46b5fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:58:16.517287 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:16.498737 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:16.517287 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:16.500280 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:16.558952 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:16.558907 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:58:16.559160 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:16.558962 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" podUID="51473c56-3874-4c1b-af5b-0601a57b1d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:58:16.646226 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:16.643308 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-xbn5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:58:16.646226 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:16.643369 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" podUID="beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:16.646226 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:58:16.644867 5057 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.165s" Oct 29 22:58:16.646226 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:16.644914 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" event={"ID":"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7","Type":"ContainerStarted","Data":"b6a2af0a6de95c74fc6c85b20c2eb2599a7743b9d62486a3ad3cfed25b58fe97"} Oct 29 22:58:16.646226 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:16.645070 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:58:16.646226 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:16.645111 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:58:16.646226 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:16.645129 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" Oct 29 22:58:16.758239 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:16.757933 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" podStartSLOduration=5.95634749 podStartE2EDuration="6.757912294s" totalImagesPullingTime="801.564804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:16.5905839 +0000 UTC m=+943.242082430" watchObservedRunningTime="2026-10-29 22:58:16.757912294 +0000 UTC m=+943.409410823" Oct 29 22:58:16.911249 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:16.887727 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 22:58:16.911249 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:16.889026 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:58:16.965280 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:16.965240 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.382955 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:17.382916 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:58:17.383272 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:17.383233 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" podUID="84648cac-9c75-47a4-a3d6-df79e46b5fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:58:17.517426 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:17.517383 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 22:58:17.522671 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:17.522627 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.540342 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:17.540305 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:58:17.540668 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:17.540634 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" podUID="51473c56-3874-4c1b-af5b-0601a57b1d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:58:17.605473 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:17.604572 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:58:17.605473 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:17.604647 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" podUID="5ac55fce-ddaa-4ffc-902d-07a374f6d967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.694238 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:17.690716 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-xbn5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:58:17.694238 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:17.690781 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" podUID="beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.892342 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:17.892293 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.976264 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:17.969504 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:18.205579 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:18.200565 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:58:18.205579 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:18.200629 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" podUID="9a4615b1-ff3d-4339-853a-f5d5247a5573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:18.378465 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:18.378373 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:58:18.378785 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:18.378751 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" podUID="84648cac-9c75-47a4-a3d6-df79e46b5fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:58:18.524424 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:18.524374 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 22:58:18.526265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:18.524971 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:18.526733 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:18.526711 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:58:18.526898 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:18.526857 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" podUID="51473c56-3874-4c1b-af5b-0601a57b1d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:58:18.713388 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:18.713349 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-xbn5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:58:18.713854 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:18.713753 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" podUID="beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:18.840857 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:18.840817 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:18.843340 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:18.843316 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:18.848648 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:18.848627 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 22:58:18.918334 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:18.918276 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:19.020269 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:19.009436 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:19.527975 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:19.527911 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:19.685392 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:19.568603 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:20.015360 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:19.980415 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:58:20.015360 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:19.987273 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" podUID="51473c56-3874-4c1b-af5b-0601a57b1d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:58:20.078286 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:20.041415 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 22:58:20.149592 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:20.132750 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 22:58:20.460270 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:20.444355 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:58:20.460270 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:20.444446 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:58:20.538820 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:20.538781 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:58:20.539427 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:20.539393 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" podUID="51473c56-3874-4c1b-af5b-0601a57b1d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:58:20.612272 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:20.611975 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:20.776271 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:20.759286 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 22:58:21.143605 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:21.143511 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:21.169972 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:21.169928 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:58:21.191893 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:21.191862 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:58:21.241214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:21.204130 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:58:21.241214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:21.204203 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" podUID="84648cac-9c75-47a4-a3d6-df79e46b5fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:21.560210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:21.552254 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:58:21.560210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:21.552329 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" podUID="51473c56-3874-4c1b-af5b-0601a57b1d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:58:21.818737 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:21.818650 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:21.821208 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:21.821158 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:21.837354 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:21.837324 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 22:58:22.514776 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:22.514551 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:58:22.514776 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:22.514631 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" podUID="51473c56-3874-4c1b-af5b-0601a57b1d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:58:22.798464 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:22.798379 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" Oct 29 22:58:23.515392 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:23.515358 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:58:23.515574 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:23.515414 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" podUID="51473c56-3874-4c1b-af5b-0601a57b1d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:58:24.593162 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:24.593129 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:24.593652 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:24.593472 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:24.594533 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:24.594513 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 22:58:31.769034 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:31.768989 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:58:33.918615 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:58:33.918570 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:58:34.117020 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:34.116990 5057 scope.go:122] "RemoveContainer" containerID="cfb445469a2a77718c1fc8a4dd1d57905009396ede3a9e6c124c592fde602f45" Oct 29 22:58:34.153337 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:34.153307 5057 scope.go:122] "RemoveContainer" containerID="2766edb18d9b887bef2b9eff2bc2f2f130e88e08d1726221e2a3b23eeae10ed4" Oct 29 22:58:34.163674 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:34.163644 5057 scope.go:122] "RemoveContainer" containerID="6d3d48385d9d0295d8d7135770340473e4352c039924e77b1e4acd6ec9eabd75" Oct 29 22:58:35.052469 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.052433 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb"] Oct 29 22:58:35.058704 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.058673 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.098586 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.098544 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb"] Oct 29 22:58:35.175726 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.175687 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.176063 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.176042 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvtbw\" (UniqueName: \"kubernetes.io/projected/c5db1599-6906-4dcd-8182-787026e2e745-kube-api-access-lvtbw\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.176255 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.176237 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.176425 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.176410 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.176591 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.176577 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5db1599-6906-4dcd-8182-787026e2e745-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.176747 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.176734 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5db1599-6906-4dcd-8182-787026e2e745-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.176909 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.176895 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.177076 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.177060 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5db1599-6906-4dcd-8182-787026e2e745-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.177389 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.177374 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.316495 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.284120 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.316826 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.316798 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.317016 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.316999 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvtbw\" (UniqueName: \"kubernetes.io/projected/c5db1599-6906-4dcd-8182-787026e2e745-kube-api-access-lvtbw\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.317129 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.317116 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.317283 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.317266 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.317414 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.317401 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5db1599-6906-4dcd-8182-787026e2e745-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.317564 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.317551 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5db1599-6906-4dcd-8182-787026e2e745-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.317713 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.317699 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.317844 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.317832 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5db1599-6906-4dcd-8182-787026e2e745-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.327874 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.327844 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.328089 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.316559 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.328484 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.328459 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.338214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.329514 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5db1599-6906-4dcd-8182-787026e2e745-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.338736 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.338712 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.406935 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.406896 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5db1599-6906-4dcd-8182-787026e2e745-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.429545 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.429507 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.431490 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.431462 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5db1599-6906-4dcd-8182-787026e2e745-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.461945 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.461913 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvtbw\" (UniqueName: \"kubernetes.io/projected/c5db1599-6906-4dcd-8182-787026e2e745-kube-api-access-lvtbw\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:35.675981 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:35.675948 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:36.267856 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:36.267813 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb"] Oct 29 22:58:36.268219 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:58:36.268004 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5db1599_6906_4dcd_8182_787026e2e745.slice/crio-4a30955dab50f8b988f4db608a74bf2950c0298f16515aecbf65f0762721802f WatchSource:0}: Error finding container 4a30955dab50f8b988f4db608a74bf2950c0298f16515aecbf65f0762721802f: Status 404 returned error can't find the container with id 4a30955dab50f8b988f4db608a74bf2950c0298f16515aecbf65f0762721802f Oct 29 22:58:36.271685 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:36.271644 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:58:36.271765 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:36.271735 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:58:36.271816 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:36.271780 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:58:37.093891 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:37.093851 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" event={"ID":"c5db1599-6906-4dcd-8182-787026e2e745","Type":"ContainerStarted","Data":"baa166cb6c5d9234be7485ef6531651d68a362859ac6d517c99c59e258748111"} Oct 29 22:58:37.094075 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:37.093916 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" event={"ID":"c5db1599-6906-4dcd-8182-787026e2e745","Type":"ContainerStarted","Data":"4a30955dab50f8b988f4db608a74bf2950c0298f16515aecbf65f0762721802f"} Oct 29 22:58:37.186364 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:37.186301 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" podStartSLOduration=2.186284807 podStartE2EDuration="2.186284807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:37.145941133 +0000 UTC m=+963.797439667" watchObservedRunningTime="2026-10-29 22:58:37.186284807 +0000 UTC m=+963.837783365" Oct 29 22:58:37.680212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:37.679623 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:37.682208 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:37.680861 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:58:37.682208 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:37.680908 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" podUID="c5db1599-6906-4dcd-8182-787026e2e745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:58:38.680210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.677006 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:58:38.680210 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.677068 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" podUID="c5db1599-6906-4dcd-8182-787026e2e745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:58:38.691683 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.691612 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8"] Oct 29 22:58:38.702334 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.702308 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.707274 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.704542 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8"] Oct 29 22:58:38.795303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.783889 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.795303 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.795176 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/352d7948-b3c7-4698-bf83-f7246a134611-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.795534 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.795346 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352d7948-b3c7-4698-bf83-f7246a134611-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.795534 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.795457 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.796129 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.795635 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.796129 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.795698 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.796129 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.795750 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.796129 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.795791 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352d7948-b3c7-4698-bf83-f7246a134611-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.796129 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.795937 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ngsp\" (UniqueName: \"kubernetes.io/projected/352d7948-b3c7-4698-bf83-f7246a134611-kube-api-access-2ngsp\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.897341 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.897305 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.897537 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.897381 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.897537 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.897408 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.897537 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.897436 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.897537 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.897460 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352d7948-b3c7-4698-bf83-f7246a134611-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.897537 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.897534 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ngsp\" (UniqueName: \"kubernetes.io/projected/352d7948-b3c7-4698-bf83-f7246a134611-kube-api-access-2ngsp\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.897799 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.897574 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.897799 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.897592 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/352d7948-b3c7-4698-bf83-f7246a134611-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.897799 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.897651 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352d7948-b3c7-4698-bf83-f7246a134611-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.898376 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.898355 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.898547 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.898533 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.898946 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.898925 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.899178 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.899148 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.899536 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.899507 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352d7948-b3c7-4698-bf83-f7246a134611-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.901648 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.901627 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352d7948-b3c7-4698-bf83-f7246a134611-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.905110 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.905090 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.966864 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.966436 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/352d7948-b3c7-4698-bf83-f7246a134611-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:38.977625 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:38.977560 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ngsp\" (UniqueName: \"kubernetes.io/projected/352d7948-b3c7-4698-bf83-f7246a134611-kube-api-access-2ngsp\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:39.073914 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:39.073583 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:39.471789 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:39.471742 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8"] Oct 29 22:58:39.485409 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:39.485376 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:58:39.485503 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:39.485447 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:58:39.485503 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:39.485478 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:58:39.677860 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:39.677548 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:58:39.677860 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:39.677609 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" podUID="c5db1599-6906-4dcd-8182-787026e2e745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:58:40.222250 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:40.220433 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" event={"ID":"352d7948-b3c7-4698-bf83-f7246a134611","Type":"ContainerStarted","Data":"0e04eea48e01c1ba168effbeb8e3e8923e7e177c6a2d9480e1f2fbd409c6871a"} Oct 29 22:58:40.222250 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:40.220501 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" event={"ID":"352d7948-b3c7-4698-bf83-f7246a134611","Type":"ContainerStarted","Data":"8f9571feb4c56c213fabc9255bd42d8a503aec82527b654ad524f807d9136954"} Oct 29 22:58:40.682213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:40.679387 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:58:40.682213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:40.679457 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" podUID="c5db1599-6906-4dcd-8182-787026e2e745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:58:41.079264 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:41.078227 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:41.082331 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:41.080683 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 29 22:58:41.082331 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:41.080740 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" podUID="352d7948-b3c7-4698-bf83-f7246a134611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 29 22:58:41.829224 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:41.826783 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:58:41.829224 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:41.826864 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" podUID="c5db1599-6906-4dcd-8182-787026e2e745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:58:42.097266 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:42.091362 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 29 22:58:42.097266 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:42.091439 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" podUID="352d7948-b3c7-4698-bf83-f7246a134611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 29 22:58:42.676970 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:42.676645 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:58:42.676970 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:42.676695 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" podUID="c5db1599-6906-4dcd-8182-787026e2e745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:58:43.074874 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:43.074791 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 29 22:58:43.074874 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:43.074850 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" podUID="352d7948-b3c7-4698-bf83-f7246a134611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 29 22:58:43.678168 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:43.677469 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:58:43.678168 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:43.677527 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" podUID="c5db1599-6906-4dcd-8182-787026e2e745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:58:44.083264 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:44.080685 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 29 22:58:44.083264 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:44.080750 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" podUID="352d7948-b3c7-4698-bf83-f7246a134611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 29 22:58:45.073982 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:45.073941 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 29 22:58:45.074202 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:45.073993 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" podUID="352d7948-b3c7-4698-bf83-f7246a134611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 29 22:58:45.120367 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:45.120334 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:45.120854 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:45.120815 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:45.122399 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:45.122344 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 22:58:45.201480 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:45.201415 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" podStartSLOduration=7.201395115 podStartE2EDuration="7.201395115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:40.369623215 +0000 UTC m=+967.021121746" watchObservedRunningTime="2026-10-29 22:58:45.201395115 +0000 UTC m=+971.852893647" Oct 29 22:58:46.089753 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:46.085601 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 29 22:58:46.089753 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:46.085667 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" podUID="352d7948-b3c7-4698-bf83-f7246a134611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 29 22:58:47.074309 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:47.074261 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 29 22:58:47.074716 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:47.074345 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" podUID="352d7948-b3c7-4698-bf83-f7246a134611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 29 22:58:47.687429 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:47.687380 5057 generic.go:356] "Generic (PLEG): container finished" podID="e3cd3c9e-4e7d-4395-81da-5f43c01467ee" containerID="0dee6d22d5168608675727d1786f2d4099b04cb69acee06543ee19eecd266452" exitCode=0 Oct 29 22:58:47.687630 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:47.687452 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" event={"ID":"e3cd3c9e-4e7d-4395-81da-5f43c01467ee","Type":"ContainerDied","Data":"0dee6d22d5168608675727d1786f2d4099b04cb69acee06543ee19eecd266452"} Oct 29 22:58:47.687954 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:47.687940 5057 scope.go:122] "RemoveContainer" containerID="0dee6d22d5168608675727d1786f2d4099b04cb69acee06543ee19eecd266452" Oct 29 22:58:47.690310 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:47.690258 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:58:47.690416 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:47.690383 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:58:47.690471 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:47.690451 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:58:48.080344 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.080263 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:48.080712 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.080695 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:48.081418 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.081399 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 22:58:48.570101 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.570067 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp"] Oct 29 22:58:48.585026 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.584998 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp" Oct 29 22:58:48.595884 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.595858 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp"] Oct 29 22:58:48.601829 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.601794 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8b57a2\"" Oct 29 22:58:48.627220 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.627168 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j"] Oct 29 22:58:48.633282 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.633253 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j" Oct 29 22:58:48.653217 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.651802 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j"] Oct 29 22:58:48.653217 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.652790 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8b57a2-dockercfg-qr4gs\"" Oct 29 22:58:48.673372 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.673342 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7nth\" (UniqueName: \"kubernetes.io/projected/b414763c-3c92-42d4-b2ff-a8514487e68c-kube-api-access-f7nth\") pod \"payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp\" (UID: \"b414763c-3c92-42d4-b2ff-a8514487e68c\") " pod="openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp" Oct 29 22:58:48.673533 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.673388 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b414763c-3c92-42d4-b2ff-a8514487e68c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp\" (UID: \"b414763c-3c92-42d4-b2ff-a8514487e68c\") " pod="openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp" Oct 29 22:58:48.673533 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.673451 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230c87a0-7758-4e6c-bb81-ebd5280d63ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j\" (UID: \"230c87a0-7758-4e6c-bb81-ebd5280d63ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j" Oct 29 22:58:48.673533 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.673524 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqzx7\" (UniqueName: \"kubernetes.io/projected/230c87a0-7758-4e6c-bb81-ebd5280d63ad-kube-api-access-sqzx7\") pod \"payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j\" (UID: \"230c87a0-7758-4e6c-bb81-ebd5280d63ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j" Oct 29 22:58:48.711209 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.709740 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" event={"ID":"e3cd3c9e-4e7d-4395-81da-5f43c01467ee","Type":"ContainerStarted","Data":"243276169f0507fa1085a3a6a2f67d16417494c469f1468227e80f3634580c93"} Oct 29 22:58:48.801213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.800892 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7nth\" (UniqueName: \"kubernetes.io/projected/b414763c-3c92-42d4-b2ff-a8514487e68c-kube-api-access-f7nth\") pod \"payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp\" (UID: \"b414763c-3c92-42d4-b2ff-a8514487e68c\") " pod="openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp" Oct 29 22:58:48.801213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.800950 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b414763c-3c92-42d4-b2ff-a8514487e68c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp\" (UID: \"b414763c-3c92-42d4-b2ff-a8514487e68c\") " pod="openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp" Oct 29 22:58:48.801213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.801014 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230c87a0-7758-4e6c-bb81-ebd5280d63ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j\" (UID: \"230c87a0-7758-4e6c-bb81-ebd5280d63ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j" Oct 29 22:58:48.801213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.801090 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqzx7\" (UniqueName: \"kubernetes.io/projected/230c87a0-7758-4e6c-bb81-ebd5280d63ad-kube-api-access-sqzx7\") pod \"payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j\" (UID: \"230c87a0-7758-4e6c-bb81-ebd5280d63ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j" Oct 29 22:58:48.807218 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.802978 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b414763c-3c92-42d4-b2ff-a8514487e68c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp\" (UID: \"b414763c-3c92-42d4-b2ff-a8514487e68c\") " pod="openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp" Oct 29 22:58:48.843004 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.842927 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7nth\" (UniqueName: \"kubernetes.io/projected/b414763c-3c92-42d4-b2ff-a8514487e68c-kube-api-access-f7nth\") pod \"payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp\" (UID: \"b414763c-3c92-42d4-b2ff-a8514487e68c\") " pod="openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp" Oct 29 22:58:48.916307 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.895265 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230c87a0-7758-4e6c-bb81-ebd5280d63ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j\" (UID: \"230c87a0-7758-4e6c-bb81-ebd5280d63ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j" Oct 29 22:58:48.916307 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.911862 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqzx7\" (UniqueName: \"kubernetes.io/projected/230c87a0-7758-4e6c-bb81-ebd5280d63ad-kube-api-access-sqzx7\") pod \"payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j\" (UID: \"230c87a0-7758-4e6c-bb81-ebd5280d63ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j" Oct 29 22:58:48.916307 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.912138 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp" Oct 29 22:58:49.002209 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:48.998940 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j" Oct 29 22:58:49.466254 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:49.457840 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:58:49.466254 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:49.458034 5057 patch_prober.go:29] interesting pod/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:58:49.466254 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:49.458079 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" podUID="e3cd3c9e-4e7d-4395-81da-5f43c01467ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 29 22:58:49.483146 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:58:49.483115 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb414763c_3c92_42d4_b2ff_a8514487e68c.slice/crio-44926cc2db0777680bf6e1ee366857b83b777b66255ea505b1e0c7919d8dbc8a WatchSource:0}: Error finding container 44926cc2db0777680bf6e1ee366857b83b777b66255ea505b1e0c7919d8dbc8a: Status 404 returned error can't find the container with id 44926cc2db0777680bf6e1ee366857b83b777b66255ea505b1e0c7919d8dbc8a Oct 29 22:58:49.487274 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:49.487250 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp"] Oct 29 22:58:49.555441 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:49.532530 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j"] Oct 29 22:58:49.719262 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:49.719130 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp" event={"ID":"b414763c-3c92-42d4-b2ff-a8514487e68c","Type":"ContainerStarted","Data":"4c561035cfce99a79d78ad87a3e36640e51676c7579eb1e3ef3c6ad0f75592b2"} Oct 29 22:58:49.719262 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:49.719233 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp" Oct 29 22:58:49.719262 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:49.719257 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp" event={"ID":"b414763c-3c92-42d4-b2ff-a8514487e68c","Type":"ContainerStarted","Data":"44926cc2db0777680bf6e1ee366857b83b777b66255ea505b1e0c7919d8dbc8a"} Oct 29 22:58:49.728296 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:49.723334 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j" event={"ID":"230c87a0-7758-4e6c-bb81-ebd5280d63ad","Type":"ContainerStarted","Data":"cfb3678c9509d8ec2314daa5d550641741fd9bc2fbf5f52fc26a8da9c7a9e056"} Oct 29 22:58:49.728296 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:49.723391 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j" event={"ID":"230c87a0-7758-4e6c-bb81-ebd5280d63ad","Type":"ContainerStarted","Data":"75686c1721b92a70d427d1764fcaf78ba42ccdb582cdfbb4e0d590ebe300f686"} Oct 29 22:58:49.728296 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:49.723451 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j" Oct 29 22:58:49.772858 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:49.772797 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp" podStartSLOduration=1.77277831 podStartE2EDuration="1.77277831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:49.76738368 +0000 UTC m=+976.418882212" watchObservedRunningTime="2026-10-29 22:58:49.77277831 +0000 UTC m=+976.424276842" Oct 29 22:58:49.813453 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:49.813395 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j" podStartSLOduration=1.8133804489999998 podStartE2EDuration="1.813380449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:49.812178231 +0000 UTC m=+976.463676761" watchObservedRunningTime="2026-10-29 22:58:49.813380449 +0000 UTC m=+976.464878983" Oct 29 22:58:50.456220 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:50.456160 5057 patch_prober.go:29] interesting pod/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:58:50.456431 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:50.456245 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" podUID="e3cd3c9e-4e7d-4395-81da-5f43c01467ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 29 22:58:51.455552 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:51.455509 5057 patch_prober.go:29] interesting pod/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:58:51.455996 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:51.455578 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" podUID="e3cd3c9e-4e7d-4395-81da-5f43c01467ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 29 22:58:52.455550 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:52.455509 5057 patch_prober.go:29] interesting pod/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:58:52.455744 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:52.455569 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" podUID="e3cd3c9e-4e7d-4395-81da-5f43c01467ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 29 22:58:52.716128 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:52.716054 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:58:53.456177 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:53.456143 5057 patch_prober.go:29] interesting pod/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:58:53.456812 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:53.456782 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" podUID="e3cd3c9e-4e7d-4395-81da-5f43c01467ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 29 22:58:54.455827 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:54.455786 5057 patch_prober.go:29] interesting pod/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:58:54.456028 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:54.455851 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" podUID="e3cd3c9e-4e7d-4395-81da-5f43c01467ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 29 22:58:55.460008 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:55.459974 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:58:55.461442 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:58:55.461416 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 22:59:00.758395 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:00.758364 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp" Oct 29 22:59:00.758828 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:00.758456 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j" Oct 29 22:59:03.915490 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:59:03.915449 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:59:04.311157 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:04.311113 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz"] Oct 29 22:59:04.315984 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:04.315959 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" Oct 29 22:59:04.352499 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:04.352463 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8b57a2-serving-cert\"" Oct 29 22:59:04.359328 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:04.359298 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz"] Oct 29 22:59:04.396872 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:04.396834 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abbf2a7f-653e-4219-8594-bc67fb660a73-maas-api-tls\") pod \"maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz\" (UID: \"abbf2a7f-653e-4219-8594-bc67fb660a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" Oct 29 22:59:04.397044 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:04.396887 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbf2a7f-653e-4219-8594-bc67fb660a73-openshift-service-ca\") pod \"maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz\" (UID: \"abbf2a7f-653e-4219-8594-bc67fb660a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" Oct 29 22:59:04.397044 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:04.397012 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkqvz\" (UniqueName: \"kubernetes.io/projected/abbf2a7f-653e-4219-8594-bc67fb660a73-kube-api-access-nkqvz\") pod \"maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz\" (UID: \"abbf2a7f-653e-4219-8594-bc67fb660a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" Oct 29 22:59:04.397152 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:04.397124 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abbf2a7f-653e-4219-8594-bc67fb660a73-metrics-certs\") pod \"maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz\" (UID: \"abbf2a7f-653e-4219-8594-bc67fb660a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" Oct 29 22:59:04.499048 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:04.499016 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abbf2a7f-653e-4219-8594-bc67fb660a73-maas-api-tls\") pod \"maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz\" (UID: \"abbf2a7f-653e-4219-8594-bc67fb660a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" Oct 29 22:59:04.499291 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:04.499056 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbf2a7f-653e-4219-8594-bc67fb660a73-openshift-service-ca\") pod \"maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz\" (UID: \"abbf2a7f-653e-4219-8594-bc67fb660a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" Oct 29 22:59:04.499291 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:04.499083 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkqvz\" (UniqueName: \"kubernetes.io/projected/abbf2a7f-653e-4219-8594-bc67fb660a73-kube-api-access-nkqvz\") pod \"maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz\" (UID: \"abbf2a7f-653e-4219-8594-bc67fb660a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" Oct 29 22:59:04.499291 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:04.499159 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abbf2a7f-653e-4219-8594-bc67fb660a73-metrics-certs\") pod \"maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz\" (UID: \"abbf2a7f-653e-4219-8594-bc67fb660a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" Oct 29 22:59:04.499890 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:04.499861 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbf2a7f-653e-4219-8594-bc67fb660a73-openshift-service-ca\") pod \"maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz\" (UID: \"abbf2a7f-653e-4219-8594-bc67fb660a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" Oct 29 22:59:04.501712 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:04.501681 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abbf2a7f-653e-4219-8594-bc67fb660a73-maas-api-tls\") pod \"maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz\" (UID: \"abbf2a7f-653e-4219-8594-bc67fb660a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" Oct 29 22:59:04.501826 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:04.501732 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abbf2a7f-653e-4219-8594-bc67fb660a73-metrics-certs\") pod \"maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz\" (UID: \"abbf2a7f-653e-4219-8594-bc67fb660a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" Oct 29 22:59:04.529261 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:04.529231 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkqvz\" (UniqueName: \"kubernetes.io/projected/abbf2a7f-653e-4219-8594-bc67fb660a73-kube-api-access-nkqvz\") pod \"maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz\" (UID: \"abbf2a7f-653e-4219-8594-bc67fb660a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" Oct 29 22:59:04.656395 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:04.652015 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" Oct 29 22:59:05.093818 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:59:05.093782 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbf2a7f_653e_4219_8594_bc67fb660a73.slice/crio-e863319854c61d08029bb71614bf123b205f15bbe6ea8a61f7ec936aa7934e83 WatchSource:0}: Error finding container e863319854c61d08029bb71614bf123b205f15bbe6ea8a61f7ec936aa7934e83: Status 404 returned error can't find the container with id e863319854c61d08029bb71614bf123b205f15bbe6ea8a61f7ec936aa7934e83 Oct 29 22:59:05.094301 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:05.093897 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz"] Oct 29 22:59:05.982834 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:05.982728 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" event={"ID":"abbf2a7f-653e-4219-8594-bc67fb660a73","Type":"ContainerStarted","Data":"a1232f2980ff965b2b67aef0fab1e485fe5892c1364ec553df3522eab1d9030c"} Oct 29 22:59:05.983003 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:05.982865 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" event={"ID":"abbf2a7f-653e-4219-8594-bc67fb660a73","Type":"ContainerStarted","Data":"e863319854c61d08029bb71614bf123b205f15bbe6ea8a61f7ec936aa7934e83"} Oct 29 22:59:05.983003 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:05.982988 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" Oct 29 22:59:06.036479 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:06.036405 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" podStartSLOduration=1.412775902 podStartE2EDuration="2.036384526s" totalImagesPullingTime="623.608624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:06.033021708 +0000 UTC m=+992.684520235" watchObservedRunningTime="2026-10-29 22:59:06.036384526 +0000 UTC m=+992.687883058" Oct 29 22:59:11.995465 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:11.995432 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" Oct 29 22:59:17.841572 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:17.841536 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk"] Oct 29 22:59:17.848707 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:17.848687 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk" Oct 29 22:59:17.866381 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:17.866354 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk"] Oct 29 22:59:17.875268 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:17.875180 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-479c2a\"" Oct 29 22:59:17.875268 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:17.875242 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-479c2a-dockercfg-448q7\"" Oct 29 22:59:17.896377 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:17.896350 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb"] Oct 29 22:59:17.901789 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:17.901768 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb" Oct 29 22:59:17.926551 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:17.926514 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb"] Oct 29 22:59:17.929939 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:17.929892 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hd86\" (UniqueName: \"kubernetes.io/projected/ee6dd65e-769c-419f-b16c-1b5abf1e879d-kube-api-access-4hd86\") pod \"payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk\" (UID: \"ee6dd65e-769c-419f-b16c-1b5abf1e879d\") " pod="openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk" Oct 29 22:59:17.930076 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:17.930010 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6dd65e-769c-419f-b16c-1b5abf1e879d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk\" (UID: \"ee6dd65e-769c-419f-b16c-1b5abf1e879d\") " pod="openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk" Oct 29 22:59:18.033891 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:18.033844 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6dd65e-769c-419f-b16c-1b5abf1e879d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk\" (UID: \"ee6dd65e-769c-419f-b16c-1b5abf1e879d\") " pod="openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk" Oct 29 22:59:18.138556 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:18.138460 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879eb585-eb10-4ddf-b447-202688d1e3e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb\" (UID: \"879eb585-eb10-4ddf-b447-202688d1e3e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb" Oct 29 22:59:18.153617 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:18.047230 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6dd65e-769c-419f-b16c-1b5abf1e879d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk\" (UID: \"ee6dd65e-769c-419f-b16c-1b5abf1e879d\") " pod="openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk" Oct 29 22:59:18.154224 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:18.154037 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldh9p\" (UniqueName: \"kubernetes.io/projected/879eb585-eb10-4ddf-b447-202688d1e3e8-kube-api-access-ldh9p\") pod \"payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb\" (UID: \"879eb585-eb10-4ddf-b447-202688d1e3e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb" Oct 29 22:59:18.159599 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:18.159559 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hd86\" (UniqueName: \"kubernetes.io/projected/ee6dd65e-769c-419f-b16c-1b5abf1e879d-kube-api-access-4hd86\") pod \"payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk\" (UID: \"ee6dd65e-769c-419f-b16c-1b5abf1e879d\") " pod="openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk" Oct 29 22:59:18.239395 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:18.239363 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hd86\" (UniqueName: \"kubernetes.io/projected/ee6dd65e-769c-419f-b16c-1b5abf1e879d-kube-api-access-4hd86\") pod \"payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk\" (UID: \"ee6dd65e-769c-419f-b16c-1b5abf1e879d\") " pod="openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk" Oct 29 22:59:18.281243 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:18.271653 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879eb585-eb10-4ddf-b447-202688d1e3e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb\" (UID: \"879eb585-eb10-4ddf-b447-202688d1e3e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb" Oct 29 22:59:18.281243 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:18.271709 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldh9p\" (UniqueName: \"kubernetes.io/projected/879eb585-eb10-4ddf-b447-202688d1e3e8-kube-api-access-ldh9p\") pod \"payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb\" (UID: \"879eb585-eb10-4ddf-b447-202688d1e3e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb" Oct 29 22:59:18.281243 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:18.273070 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879eb585-eb10-4ddf-b447-202688d1e3e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb\" (UID: \"879eb585-eb10-4ddf-b447-202688d1e3e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb" Oct 29 22:59:18.357314 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:18.357286 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldh9p\" (UniqueName: \"kubernetes.io/projected/879eb585-eb10-4ddf-b447-202688d1e3e8-kube-api-access-ldh9p\") pod \"payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb\" (UID: \"879eb585-eb10-4ddf-b447-202688d1e3e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb" Oct 29 22:59:18.466716 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:18.466681 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk" Oct 29 22:59:18.522720 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:18.522462 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb" Oct 29 22:59:18.803932 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:59:18.803899 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee6dd65e_769c_419f_b16c_1b5abf1e879d.slice/crio-528cb7dd0760d8ac7c243fb109eb924bc770d0e09088f50dfbaecc24279b1508 WatchSource:0}: Error finding container 528cb7dd0760d8ac7c243fb109eb924bc770d0e09088f50dfbaecc24279b1508: Status 404 returned error can't find the container with id 528cb7dd0760d8ac7c243fb109eb924bc770d0e09088f50dfbaecc24279b1508 Oct 29 22:59:18.804259 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:18.804234 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk"] Oct 29 22:59:18.904316 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:18.904289 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb"] Oct 29 22:59:18.906290 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:59:18.906264 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879eb585_eb10_4ddf_b447_202688d1e3e8.slice/crio-ccd74884992823cfc88d4c41704c02eb12b263e17d4ed5530a86169a85a06831 WatchSource:0}: Error finding container ccd74884992823cfc88d4c41704c02eb12b263e17d4ed5530a86169a85a06831: Status 404 returned error can't find the container with id ccd74884992823cfc88d4c41704c02eb12b263e17d4ed5530a86169a85a06831 Oct 29 22:59:19.246515 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:19.246473 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb" event={"ID":"879eb585-eb10-4ddf-b447-202688d1e3e8","Type":"ContainerStarted","Data":"734c2217620f2d9e5a7df448e80f947724d887a61d657b71241ac3dea16c5ab0"} Oct 29 22:59:19.246735 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:19.246577 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb" event={"ID":"879eb585-eb10-4ddf-b447-202688d1e3e8","Type":"ContainerStarted","Data":"ccd74884992823cfc88d4c41704c02eb12b263e17d4ed5530a86169a85a06831"} Oct 29 22:59:19.246735 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:19.246609 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb" Oct 29 22:59:19.248352 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:19.248327 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk" event={"ID":"ee6dd65e-769c-419f-b16c-1b5abf1e879d","Type":"ContainerStarted","Data":"246bd1582a608d369cc7a6d923d44c464705ddd77d6c30eb6ffffb735a4d1a51"} Oct 29 22:59:19.248458 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:19.248368 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk" event={"ID":"ee6dd65e-769c-419f-b16c-1b5abf1e879d","Type":"ContainerStarted","Data":"528cb7dd0760d8ac7c243fb109eb924bc770d0e09088f50dfbaecc24279b1508"} Oct 29 22:59:19.248458 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:19.248388 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk" Oct 29 22:59:19.296688 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:19.296639 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb" podStartSLOduration=2.296625917 podStartE2EDuration="2.296625917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:19.292792195 +0000 UTC m=+1005.944290724" watchObservedRunningTime="2026-10-29 22:59:19.296625917 +0000 UTC m=+1005.948124443" Oct 29 22:59:19.338315 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:19.338264 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk" podStartSLOduration=2.3382506960000002 podStartE2EDuration="2.338250696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:19.336866092 +0000 UTC m=+1005.988364621" watchObservedRunningTime="2026-10-29 22:59:19.338250696 +0000 UTC m=+1005.989749226" Oct 29 22:59:27.917719 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:27.917687 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w"] Oct 29 22:59:27.928896 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:27.928865 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w" Oct 29 22:59:27.943573 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:27.943545 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w"] Oct 29 22:59:27.951315 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:27.951287 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-660540-dockercfg-29vc2\"" Oct 29 22:59:27.951432 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:27.951391 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-660540\"" Oct 29 22:59:27.993813 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:27.993775 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8fe269-cf2b-45ae-8965-640b6404a618-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w\" (UID: \"ff8fe269-cf2b-45ae-8965-640b6404a618\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w" Oct 29 22:59:27.993990 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:27.993950 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frbgm\" (UniqueName: \"kubernetes.io/projected/ff8fe269-cf2b-45ae-8965-640b6404a618-kube-api-access-frbgm\") pod \"payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w\" (UID: \"ff8fe269-cf2b-45ae-8965-640b6404a618\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w" Oct 29 22:59:28.114213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:28.110036 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frbgm\" (UniqueName: \"kubernetes.io/projected/ff8fe269-cf2b-45ae-8965-640b6404a618-kube-api-access-frbgm\") pod \"payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w\" (UID: \"ff8fe269-cf2b-45ae-8965-640b6404a618\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w" Oct 29 22:59:28.114213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:28.110420 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8fe269-cf2b-45ae-8965-640b6404a618-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w\" (UID: \"ff8fe269-cf2b-45ae-8965-640b6404a618\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w" Oct 29 22:59:28.240223 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:28.218901 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8fe269-cf2b-45ae-8965-640b6404a618-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w\" (UID: \"ff8fe269-cf2b-45ae-8965-640b6404a618\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w" Oct 29 22:59:28.264212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:28.242956 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frbgm\" (UniqueName: \"kubernetes.io/projected/ff8fe269-cf2b-45ae-8965-640b6404a618-kube-api-access-frbgm\") pod \"payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w\" (UID: \"ff8fe269-cf2b-45ae-8965-640b6404a618\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w" Oct 29 22:59:28.550236 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:28.550109 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w" Oct 29 22:59:28.860182 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:28.860152 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w"] Oct 29 22:59:28.861409 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:59:28.861380 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8fe269_cf2b_45ae_8965_640b6404a618.slice/crio-66db97bce45b9c1a08a2e07afd6ea4ed00ec0f7ee4bb7809896681f8331f1203 WatchSource:0}: Error finding container 66db97bce45b9c1a08a2e07afd6ea4ed00ec0f7ee4bb7809896681f8331f1203: Status 404 returned error can't find the container with id 66db97bce45b9c1a08a2e07afd6ea4ed00ec0f7ee4bb7809896681f8331f1203 Oct 29 22:59:29.448647 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:29.448594 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w" event={"ID":"ff8fe269-cf2b-45ae-8965-640b6404a618","Type":"ContainerStarted","Data":"d9e0f42318f8a44457cbb000c26322eea2c5881974cf6e5b1f05d3e0a6dc5111"} Oct 29 22:59:29.449137 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:29.448704 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w" event={"ID":"ff8fe269-cf2b-45ae-8965-640b6404a618","Type":"ContainerStarted","Data":"66db97bce45b9c1a08a2e07afd6ea4ed00ec0f7ee4bb7809896681f8331f1203"} Oct 29 22:59:29.449137 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:29.448740 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w" Oct 29 22:59:29.495902 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:29.495854 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w" podStartSLOduration=2.495839994 podStartE2EDuration="2.495839994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:29.493348235 +0000 UTC m=+1016.144846765" watchObservedRunningTime="2026-10-29 22:59:29.495839994 +0000 UTC m=+1016.147338556" Oct 29 22:59:30.265268 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:30.265232 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk" Oct 29 22:59:30.265473 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:30.265342 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb" Oct 29 22:59:31.023792 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.023722 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd"] Oct 29 22:59:31.031458 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.031429 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.054211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.053004 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd"] Oct 29 22:59:31.299892 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.299790 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwfg9\" (UniqueName: \"kubernetes.io/projected/e794e7ae-93a6-4789-9542-217137d0aa1a-kube-api-access-lwfg9\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.304214 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.300396 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.331770 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.331720 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e794e7ae-93a6-4789-9542-217137d0aa1a-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.396339 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.391023 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 22:59:31.408301 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.404427 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.428873 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.428832 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.440544 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.440429 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.446343 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.446309 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.448226 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.448171 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.450694 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.450036 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.552313 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.552218 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e794e7ae-93a6-4789-9542-217137d0aa1a-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.552491 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.552346 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.552491 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.552442 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.552615 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.552602 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.552670 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.552647 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.552749 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.552723 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.552817 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.552764 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.552817 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.552790 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.552915 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.552876 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwfg9\" (UniqueName: \"kubernetes.io/projected/e794e7ae-93a6-4789-9542-217137d0aa1a-kube-api-access-lwfg9\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.552973 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.552940 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.553522 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.553043 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.553522 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.553093 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e794e7ae-93a6-4789-9542-217137d0aa1a-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.553716 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.553643 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.553818 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.553784 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.556033 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.556006 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.556224 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.556171 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.578983 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.578934 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.579258 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.579220 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwfg9\" (UniqueName: \"kubernetes.io/projected/e794e7ae-93a6-4789-9542-217137d0aa1a-kube-api-access-lwfg9\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.615953 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.615925 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:31.947727 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:59:31.947684 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode794e7ae_93a6_4789_9542_217137d0aa1a.slice/crio-164f036696e270d5516c3112360e8fbad396217da5645220302287c0f3c3fac3 WatchSource:0}: Error finding container 164f036696e270d5516c3112360e8fbad396217da5645220302287c0f3c3fac3: Status 404 returned error can't find the container with id 164f036696e270d5516c3112360e8fbad396217da5645220302287c0f3c3fac3 Oct 29 22:59:31.948104 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.948086 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd"] Oct 29 22:59:31.950498 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.950464 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:59:31.950572 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.950542 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:59:31.950611 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:31.950570 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:59:32.502212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:32.493664 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" event={"ID":"e794e7ae-93a6-4789-9542-217137d0aa1a","Type":"ContainerStarted","Data":"9c7ee8762d549d259c59d7f7f1b74f5eb9bc58dbdc0472a790e08a3809a2dacc"} Oct 29 22:59:32.502212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:32.493774 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" event={"ID":"e794e7ae-93a6-4789-9542-217137d0aa1a","Type":"ContainerStarted","Data":"164f036696e270d5516c3112360e8fbad396217da5645220302287c0f3c3fac3"} Oct 29 22:59:32.564153 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:32.564093 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" podStartSLOduration=2.564071221 podStartE2EDuration="2.564071221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:32.557302849 +0000 UTC m=+1019.208801371" watchObservedRunningTime="2026-10-29 22:59:32.564071221 +0000 UTC m=+1019.215569752" Oct 29 22:59:32.616906 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:32.616875 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:33.619231 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:33.618959 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:59:33.619231 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:33.619004 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" podUID="e794e7ae-93a6-4789-9542-217137d0aa1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:59:34.018800 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:59:34.018729 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 22:59:34.621237 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:34.618559 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:59:34.621237 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:34.618618 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" podUID="e794e7ae-93a6-4789-9542-217137d0aa1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:59:35.660211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:35.655824 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:59:35.660211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:35.655885 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" podUID="e794e7ae-93a6-4789-9542-217137d0aa1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:59:35.935391 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:35.896502 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:59:35.935391 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:35.916406 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:59:36.002222 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:35.987265 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:59:36.542211 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:36.541720 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 22:59:36.550378 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:36.550330 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:59:36.632280 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:36.632242 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:59:36.632528 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:36.632499 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" podUID="e794e7ae-93a6-4789-9542-217137d0aa1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:59:36.922372 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:36.922275 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:59:36.923796 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:36.923772 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:59:37.017874 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:37.016404 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:59:37.019074 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:37.019040 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:59:37.552250 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:37.547547 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:59:37.553342 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:37.553308 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 22:59:37.616708 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:37.616671 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:59:37.616968 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:37.616939 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" podUID="e794e7ae-93a6-4789-9542-217137d0aa1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:59:37.930711 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:37.930665 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 22:59:38.064264 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:38.020285 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:59:38.230293 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:38.230057 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:59:38.230293 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:38.230130 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:59:38.281985 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:38.272333 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 22:59:38.436223 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:38.433474 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 22:59:38.436223 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:38.435262 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 22:59:38.457652 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:38.455474 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 22:59:38.617908 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:38.617819 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:59:38.617908 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:38.617880 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" podUID="e794e7ae-93a6-4789-9542-217137d0aa1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:59:38.965648 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:38.965601 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9"] Oct 29 22:59:39.141290 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:39.141240 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk"] Oct 29 22:59:39.623705 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:39.622518 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:59:39.623705 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:39.622578 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" podUID="e794e7ae-93a6-4789-9542-217137d0aa1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:59:40.478365 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:40.478332 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w" Oct 29 22:59:40.617669 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:40.617633 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:59:40.617847 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:40.617691 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" podUID="e794e7ae-93a6-4789-9542-217137d0aa1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:59:42.267105 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.267069 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb"] Oct 29 22:59:42.292290 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.287927 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.311355 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.311302 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-475876-openshift-default-dockercfg-ms48w\"" Oct 29 22:59:42.317668 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.317591 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb"] Oct 29 22:59:42.447212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.441661 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-credential-socket\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.447212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.441743 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-istio-envoy\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.447212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.441811 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860644cd-397b-4af8-b425-a81491c9b66b-istio-podinfo\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.447212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.441931 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg49f\" (UniqueName: \"kubernetes.io/projected/860644cd-397b-4af8-b425-a81491c9b66b-kube-api-access-pg49f\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.447212 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.441998 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-workload-certs\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.488233 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.464313 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-istio-data\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.488233 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.464376 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860644cd-397b-4af8-b425-a81491c9b66b-istiod-ca-cert\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.488233 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.464455 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-workload-socket\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.488233 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.464556 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860644cd-397b-4af8-b425-a81491c9b66b-istio-token\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.567265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.566275 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860644cd-397b-4af8-b425-a81491c9b66b-istio-token\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.567265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.566358 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-credential-socket\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.567265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.566449 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-istio-envoy\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.567265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.566545 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860644cd-397b-4af8-b425-a81491c9b66b-istio-podinfo\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.567265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.566748 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg49f\" (UniqueName: \"kubernetes.io/projected/860644cd-397b-4af8-b425-a81491c9b66b-kube-api-access-pg49f\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.567265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.566846 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-workload-certs\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.567265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.566945 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-istio-data\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.567265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.566979 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860644cd-397b-4af8-b425-a81491c9b66b-istiod-ca-cert\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.567265 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.567065 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-workload-socket\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.567863 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.567554 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-workload-socket\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.571286 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.568348 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-credential-socket\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.583241 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.579472 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-istio-envoy\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.602793 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.602753 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-istio-data\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.605758 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.603437 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860644cd-397b-4af8-b425-a81491c9b66b-istiod-ca-cert\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.607298 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.604051 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-workload-certs\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.607460 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.605631 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860644cd-397b-4af8-b425-a81491c9b66b-istio-podinfo\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.626324 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.626295 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860644cd-397b-4af8-b425-a81491c9b66b-istio-token\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.649221 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.634012 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:59:42.651077 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.651035 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" podUID="e794e7ae-93a6-4789-9542-217137d0aa1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:59:42.665157 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.664922 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg49f\" (UniqueName: \"kubernetes.io/projected/860644cd-397b-4af8-b425-a81491c9b66b-kube-api-access-pg49f\") pod \"e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:42.936801 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:42.936767 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:43.493474 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:43.493338 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb"] Oct 29 22:59:43.495443 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:43.495407 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:59:43.495558 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:43.495486 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:59:43.495558 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:43.495533 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:59:43.693551 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:43.693509 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:59:43.693719 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:43.693585 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" podUID="e794e7ae-93a6-4789-9542-217137d0aa1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:59:43.891529 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:43.891445 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:43.893591 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:43.893563 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:43.895398 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:43.895378 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 22:59:43.966049 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:43.966007 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9" podUID="998b1e6e-e9fc-493b-aca5-2cf89f43cc04" containerName="payload-processing" containerID="cri-o://9ca5bb60d789b5c89f9c856897cc5fccb615ee21def3a3745f409bbf16e0d48e" gracePeriod=25 Oct 29 22:59:44.154868 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.154774 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk" podUID="7b24c9e7-cc9d-4f56-a440-aaec87ee199b" containerName="payload-pre-processing" containerID="cri-o://98b67451818b26d1b75f18b1fa0d5a92a02497a7676a5c6a355b4a1ec341737f" gracePeriod=25 Oct 29 22:59:44.306001 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.305957 5057 generic.go:356] "Generic (PLEG): container finished" podID="7b24c9e7-cc9d-4f56-a440-aaec87ee199b" containerID="98b67451818b26d1b75f18b1fa0d5a92a02497a7676a5c6a355b4a1ec341737f" exitCode=0 Oct 29 22:59:44.306222 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.306163 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk" event={"ID":"7b24c9e7-cc9d-4f56-a440-aaec87ee199b","Type":"ContainerDied","Data":"98b67451818b26d1b75f18b1fa0d5a92a02497a7676a5c6a355b4a1ec341737f"} Oct 29 22:59:44.308255 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.308222 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" event={"ID":"860644cd-397b-4af8-b425-a81491c9b66b","Type":"ContainerStarted","Data":"0c141101a46c5d3ae43fd187abb8deabb90fa5e28d66d5dbc51f1096d107cd7f"} Oct 29 22:59:44.308389 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.308279 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" event={"ID":"860644cd-397b-4af8-b425-a81491c9b66b","Type":"ContainerStarted","Data":"bb55a90499dbdee488a69fedb3a886ee50131abcde80cdfb9c02582dba92e654"} Oct 29 22:59:44.309766 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.309743 5057 generic.go:356] "Generic (PLEG): container finished" podID="998b1e6e-e9fc-493b-aca5-2cf89f43cc04" containerID="9ca5bb60d789b5c89f9c856897cc5fccb615ee21def3a3745f409bbf16e0d48e" exitCode=0 Oct 29 22:59:44.309869 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.309824 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9" event={"ID":"998b1e6e-e9fc-493b-aca5-2cf89f43cc04","Type":"ContainerDied","Data":"9ca5bb60d789b5c89f9c856897cc5fccb615ee21def3a3745f409bbf16e0d48e"} Oct 29 22:59:44.309909 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.309867 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9" event={"ID":"998b1e6e-e9fc-493b-aca5-2cf89f43cc04","Type":"ContainerDied","Data":"006005ac9f155739e1b69c7a1f12baff4ecdce67710c03857b1169ac232558e9"} Oct 29 22:59:44.309909 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.309889 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="006005ac9f155739e1b69c7a1f12baff4ecdce67710c03857b1169ac232558e9" Oct 29 22:59:44.324618 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.324596 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9" Oct 29 22:59:44.347748 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.347714 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998b1e6e-e9fc-493b-aca5-2cf89f43cc04-kube-api-access-gbsfv\" (UniqueName: \"kubernetes.io/projected/998b1e6e-e9fc-493b-aca5-2cf89f43cc04-kube-api-access-gbsfv\") pod \"998b1e6e-e9fc-493b-aca5-2cf89f43cc04\" (UID: \"998b1e6e-e9fc-493b-aca5-2cf89f43cc04\") " Oct 29 22:59:44.348035 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.348020 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/998b1e6e-e9fc-493b-aca5-2cf89f43cc04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998b1e6e-e9fc-493b-aca5-2cf89f43cc04-plugins-config-volume\") pod \"998b1e6e-e9fc-493b-aca5-2cf89f43cc04\" (UID: \"998b1e6e-e9fc-493b-aca5-2cf89f43cc04\") " Oct 29 22:59:44.349227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.349133 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998b1e6e-e9fc-493b-aca5-2cf89f43cc04-plugins-config-volume" pod "998b1e6e-e9fc-493b-aca5-2cf89f43cc04" (UID: "998b1e6e-e9fc-493b-aca5-2cf89f43cc04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:44.350680 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.350654 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998b1e6e-e9fc-493b-aca5-2cf89f43cc04-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:59:44.357720 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.357684 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998b1e6e-e9fc-493b-aca5-2cf89f43cc04-kube-api-access-gbsfv" pod "998b1e6e-e9fc-493b-aca5-2cf89f43cc04" (UID: "998b1e6e-e9fc-493b-aca5-2cf89f43cc04"). InnerVolumeSpecName "kube-api-access-gbsfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:44.358571 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.358521 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" podStartSLOduration=2.358504606 podStartE2EDuration="2.358504606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:44.357971109 +0000 UTC m=+1031.009469640" watchObservedRunningTime="2026-10-29 22:59:44.358504606 +0000 UTC m=+1031.010003136" Oct 29 22:59:44.443558 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.443531 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk" Oct 29 22:59:44.452520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.452493 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbsfv\" (UniqueName: \"kubernetes.io/projected/998b1e6e-e9fc-493b-aca5-2cf89f43cc04-kube-api-access-gbsfv\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:59:44.701711 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.701319 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b24c9e7-cc9d-4f56-a440-aaec87ee199b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b24c9e7-cc9d-4f56-a440-aaec87ee199b-plugins-config-volume\") pod \"7b24c9e7-cc9d-4f56-a440-aaec87ee199b\" (UID: \"7b24c9e7-cc9d-4f56-a440-aaec87ee199b\") " Oct 29 22:59:44.702383 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.702358 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b24c9e7-cc9d-4f56-a440-aaec87ee199b-kube-api-access-ffvkb\" (UniqueName: \"kubernetes.io/projected/7b24c9e7-cc9d-4f56-a440-aaec87ee199b-kube-api-access-ffvkb\") pod \"7b24c9e7-cc9d-4f56-a440-aaec87ee199b\" (UID: \"7b24c9e7-cc9d-4f56-a440-aaec87ee199b\") " Oct 29 22:59:44.705411 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.705384 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b24c9e7-cc9d-4f56-a440-aaec87ee199b-plugins-config-volume" pod "7b24c9e7-cc9d-4f56-a440-aaec87ee199b" (UID: "7b24c9e7-cc9d-4f56-a440-aaec87ee199b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:44.707169 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.707145 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b24c9e7-cc9d-4f56-a440-aaec87ee199b-kube-api-access-ffvkb" pod "7b24c9e7-cc9d-4f56-a440-aaec87ee199b" (UID: "7b24c9e7-cc9d-4f56-a440-aaec87ee199b"). InnerVolumeSpecName "kube-api-access-ffvkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:44.805415 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.805383 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffvkb\" (UniqueName: \"kubernetes.io/projected/7b24c9e7-cc9d-4f56-a440-aaec87ee199b-kube-api-access-ffvkb\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:59:44.805415 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.805409 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b24c9e7-cc9d-4f56-a440-aaec87ee199b-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 22:59:44.938213 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.937608 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:44.974368 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:44.969524 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:45.319324 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:45.319236 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk" Oct 29 22:59:45.319324 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:45.319224 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk" event={"ID":"7b24c9e7-cc9d-4f56-a440-aaec87ee199b","Type":"ContainerDied","Data":"4ca174e1582c0e310b4c830dc8019a10f44c11d0d63b28b91a4f92441d2acee0"} Oct 29 22:59:45.319520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:45.319366 5057 scope.go:122] "RemoveContainer" containerID="98b67451818b26d1b75f18b1fa0d5a92a02497a7676a5c6a355b4a1ec341737f" Oct 29 22:59:45.319520 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:45.319423 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9" Oct 29 22:59:45.320116 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:45.320089 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:45.321240 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:45.321180 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 22:59:45.428609 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:45.428568 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk"] Oct 29 22:59:45.436310 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:45.436278 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c79a4e-7c576b78dc-2xfpk"] Oct 29 22:59:45.484758 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:45.484722 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b24c9e7-cc9d-4f56-a440-aaec87ee199b" path="/var/lib/kubelet/pods/7b24c9e7-cc9d-4f56-a440-aaec87ee199b/volumes" Oct 29 22:59:45.485076 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:45.485062 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9"] Oct 29 22:59:45.485139 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:45.485092 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c79a4e-dbf55d956-6dfh9"] Oct 29 22:59:47.483493 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:47.483420 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="998b1e6e-e9fc-493b-aca5-2cf89f43cc04" path="/var/lib/kubelet/pods/998b1e6e-e9fc-493b-aca5-2cf89f43cc04/volumes" Oct 29 22:59:48.684869 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:48.684836 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c"] Oct 29 22:59:48.685620 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:48.685606 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7b24c9e7-cc9d-4f56-a440-aaec87ee199b" containerName="payload-pre-processing" Oct 29 22:59:48.685712 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:48.685621 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b24c9e7-cc9d-4f56-a440-aaec87ee199b" containerName="payload-pre-processing" Oct 29 22:59:48.685712 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:48.685655 5057 cpu_manager.go:414] "containerMap: removing container" podUID="998b1e6e-e9fc-493b-aca5-2cf89f43cc04" containerName="payload-processing" Oct 29 22:59:48.685712 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:48.685660 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998b1e6e-e9fc-493b-aca5-2cf89f43cc04" containerName="payload-processing" Oct 29 22:59:48.685817 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:48.685755 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b24c9e7-cc9d-4f56-a440-aaec87ee199b" containerName="payload-pre-processing" Oct 29 22:59:48.685817 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:48.685770 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="998b1e6e-e9fc-493b-aca5-2cf89f43cc04" containerName="payload-processing" Oct 29 22:59:48.691118 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:48.691099 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c" Oct 29 22:59:48.712474 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:48.712445 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c"] Oct 29 22:59:48.726054 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:48.726023 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-475876\"" Oct 29 22:59:48.726243 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:48.726213 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-475876-dockercfg-lq9jn\"" Oct 29 22:59:49.065267 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:49.000033 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e373476e-bc93-4405-b912-ba19c6636462-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c\" (UID: \"e373476e-bc93-4405-b912-ba19c6636462\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c" Oct 29 22:59:49.163617 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:49.068574 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t26l5\" (UniqueName: \"kubernetes.io/projected/e373476e-bc93-4405-b912-ba19c6636462-kube-api-access-t26l5\") pod \"payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c\" (UID: \"e373476e-bc93-4405-b912-ba19c6636462\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c" Oct 29 22:59:49.240542 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:49.238259 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t26l5\" (UniqueName: \"kubernetes.io/projected/e373476e-bc93-4405-b912-ba19c6636462-kube-api-access-t26l5\") pod \"payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c\" (UID: \"e373476e-bc93-4405-b912-ba19c6636462\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c" Oct 29 22:59:49.240542 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:49.238427 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e373476e-bc93-4405-b912-ba19c6636462-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c\" (UID: \"e373476e-bc93-4405-b912-ba19c6636462\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c" Oct 29 22:59:49.293316 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:49.264408 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e373476e-bc93-4405-b912-ba19c6636462-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c\" (UID: \"e373476e-bc93-4405-b912-ba19c6636462\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c" Oct 29 22:59:49.315227 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:49.311801 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t26l5\" (UniqueName: \"kubernetes.io/projected/e373476e-bc93-4405-b912-ba19c6636462-kube-api-access-t26l5\") pod \"payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c\" (UID: \"e373476e-bc93-4405-b912-ba19c6636462\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c" Oct 29 22:59:49.321765 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:49.321680 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c" Oct 29 22:59:49.785128 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:59:49.785092 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode373476e_bc93_4405_b912_ba19c6636462.slice/crio-b4e66f73203899fd8f94f91f4cb9fade2c59673198bd2363cc81b586359362f7 WatchSource:0}: Error finding container b4e66f73203899fd8f94f91f4cb9fade2c59673198bd2363cc81b586359362f7: Status 404 returned error can't find the container with id b4e66f73203899fd8f94f91f4cb9fade2c59673198bd2363cc81b586359362f7 Oct 29 22:59:49.785607 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:49.785567 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c"] Oct 29 22:59:50.502535 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:50.502496 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c" event={"ID":"e373476e-bc93-4405-b912-ba19c6636462","Type":"ContainerStarted","Data":"8e96911d78fd28730c08e1c3fc31fe2945f761d66970b160ffbff6cd8bca4670"} Oct 29 22:59:50.502718 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:50.502564 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c" event={"ID":"e373476e-bc93-4405-b912-ba19c6636462","Type":"ContainerStarted","Data":"b4e66f73203899fd8f94f91f4cb9fade2c59673198bd2363cc81b586359362f7"} Oct 29 22:59:50.502718 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:50.502596 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c" Oct 29 22:59:50.580209 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:50.580146 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c" podStartSLOduration=2.5801322410000003 podStartE2EDuration="2.580132241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:50.576438705 +0000 UTC m=+1037.227937234" watchObservedRunningTime="2026-10-29 22:59:50.580132241 +0000 UTC m=+1037.231630781" Oct 29 22:59:51.059098 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.059061 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m"] Oct 29 22:59:51.064890 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.064867 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" Oct 29 22:59:51.085161 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.085122 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m"] Oct 29 22:59:51.125556 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.125521 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-660540-serving-cert\"" Oct 29 22:59:51.126809 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.126777 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr"] Oct 29 22:59:51.160155 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.160115 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdgpn\" (UniqueName: \"kubernetes.io/projected/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-kube-api-access-hdgpn\") pod \"maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m\" (UID: \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" Oct 29 22:59:51.160583 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.160547 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-metrics-certs\") pod \"maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m\" (UID: \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" Oct 29 22:59:51.160850 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.160829 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m\" (UID: \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" Oct 29 22:59:51.161328 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.161307 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-maas-api-tls\") pod \"maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m\" (UID: \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" Oct 29 22:59:51.161841 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.161812 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr"] Oct 29 22:59:51.162147 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.162123 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.187301 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.187274 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-df82ee-openshift-default-dockercfg-k9pl9\"" Oct 29 22:59:51.263403 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.263367 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdgpn\" (UniqueName: \"kubernetes.io/projected/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-kube-api-access-hdgpn\") pod \"maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m\" (UID: \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" Oct 29 22:59:51.263673 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.263643 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-workload-certs\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.263811 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.263690 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-workload-socket\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.263811 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.263738 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-metrics-certs\") pod \"maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m\" (UID: \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" Oct 29 22:59:51.263940 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.263806 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-token\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.263995 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.263973 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m\" (UID: \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" Oct 29 22:59:51.264065 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.264047 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq6hp\" (UniqueName: \"kubernetes.io/projected/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-kube-api-access-kq6hp\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.264199 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.264165 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-podinfo\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.264318 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.264231 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-credential-socket\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.264383 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.264317 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-data\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.264540 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.264524 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-maas-api-tls\") pod \"maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m\" (UID: \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" Oct 29 22:59:51.264623 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.264600 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istiod-ca-cert\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.264684 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.264650 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-envoy\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.264926 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:59:51.264895 5057 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-serving-cert: secret "maas-api-e2e-ipp-a-660540-serving-cert" not found Oct 29 22:59:51.265120 ip-10-0-135-80 kubenswrapper[5057]: E1029 22:59:51.265101 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-maas-api-tls podName:6014cb97-2d6c-4ac3-b24c-e7612e0cc900 nodeName:}" failed. No retries permitted until 2026-10-29 22:59:51.765071427 +0000 UTC m=+1038.416569950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-maas-api-tls") pod "maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" (UID: "6014cb97-2d6c-4ac3-b24c-e7612e0cc900") : secret "maas-api-e2e-ipp-a-660540-serving-cert" not found Oct 29 22:59:51.265770 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.265745 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m\" (UID: \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" Oct 29 22:59:51.266978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.266947 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-metrics-certs\") pod \"maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m\" (UID: \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" Oct 29 22:59:51.288643 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.288619 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdgpn\" (UniqueName: \"kubernetes.io/projected/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-kube-api-access-hdgpn\") pod \"maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m\" (UID: \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" Oct 29 22:59:51.384262 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.383071 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-podinfo\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.384262 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.383125 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-credential-socket\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.384262 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.383210 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-data\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.384262 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.383412 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istiod-ca-cert\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.384262 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.383456 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-envoy\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.384262 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.383562 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-workload-certs\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.384262 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.383603 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-workload-socket\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.384262 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.383647 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-token\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.384262 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.383735 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq6hp\" (UniqueName: \"kubernetes.io/projected/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-kube-api-access-kq6hp\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.385531 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.385505 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-workload-socket\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.389198 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.385919 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-workload-certs\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.399283 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.386347 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-data\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.399467 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.386569 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-credential-socket\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.399575 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.387499 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istiod-ca-cert\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.432208 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.428487 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-envoy\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.476231 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.463295 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq6hp\" (UniqueName: \"kubernetes.io/projected/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-kube-api-access-kq6hp\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.477052 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.477022 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-podinfo\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.585277 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.553670 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-token\") pod \"e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.786260 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.785932 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:51.846296 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.846259 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-maas-api-tls\") pod \"maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m\" (UID: \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" Oct 29 22:59:51.849602 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.849550 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-maas-api-tls\") pod \"maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m\" (UID: \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" Oct 29 22:59:51.989235 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:51.986588 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" Oct 29 22:59:52.137966 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:52.137849 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr"] Oct 29 22:59:52.141376 ip-10-0-135-80 kubenswrapper[5057]: W1029 22:59:52.141347 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f124749_58f1_491c_b3e8_f1a0d0adb8dc.slice/crio-20fdbea70e3e36f36f172b53a2fe3221bbbb6179a7ef7fb6bd0d6a23d42867e1 WatchSource:0}: Error finding container 20fdbea70e3e36f36f172b53a2fe3221bbbb6179a7ef7fb6bd0d6a23d42867e1: Status 404 returned error can't find the container with id 20fdbea70e3e36f36f172b53a2fe3221bbbb6179a7ef7fb6bd0d6a23d42867e1 Oct 29 22:59:52.145401 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:52.145361 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:59:52.145577 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:52.145557 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:59:52.145709 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:52.145692 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 22:59:52.500879 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:52.500833 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m"] Oct 29 22:59:52.708978 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:52.708943 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" event={"ID":"6014cb97-2d6c-4ac3-b24c-e7612e0cc900","Type":"ContainerStarted","Data":"5f4c65c8a85e1631139b41ddf99bcd5c813e65552e33aa70463b9b957ca8baf6"} Oct 29 22:59:52.710865 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:52.710830 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" event={"ID":"9f124749-58f1-491c-b3e8-f1a0d0adb8dc","Type":"ContainerStarted","Data":"82182bc5029c288b02f5fbeeb6f83d83d5efc4d8e3568e39950f8c699877af13"} Oct 29 22:59:52.711009 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:52.710880 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" event={"ID":"9f124749-58f1-491c-b3e8-f1a0d0adb8dc","Type":"ContainerStarted","Data":"20fdbea70e3e36f36f172b53a2fe3221bbbb6179a7ef7fb6bd0d6a23d42867e1"} Oct 29 22:59:52.761959 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:52.761853 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" podStartSLOduration=1.7618348579999998 podStartE2EDuration="1.761834858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:52.759181403 +0000 UTC m=+1039.410679932" watchObservedRunningTime="2026-10-29 22:59:52.761834858 +0000 UTC m=+1039.413333389" Oct 29 22:59:52.787296 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:52.787266 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:53.719633 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:53.719589 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" event={"ID":"6014cb97-2d6c-4ac3-b24c-e7612e0cc900","Type":"ContainerStarted","Data":"ccd917add18bb5d44edef0a798f569d0984ecc993ef8a95c37e1108ce601cc02"} Oct 29 22:59:53.720027 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:53.719661 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" Oct 29 22:59:53.771055 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:53.770990 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" podStartSLOduration=2.428956597 podStartE2EDuration="2.770971659s" totalImagesPullingTime="342.015062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:53.76566868 +0000 UTC m=+1040.417167212" watchObservedRunningTime="2026-10-29 22:59:53.770971659 +0000 UTC m=+1040.422470190" Oct 29 22:59:53.793216 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:53.793176 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:54.729123 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:54.729090 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:54.730230 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:54.730177 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 22:59:58.406276 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:58.404973 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2"] Oct 29 22:59:59.736009 ip-10-0-135-80 kubenswrapper[5057]: I1029 22:59:59.735975 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" Oct 29 23:00:00.230024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:00.229984 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5"] Oct 29 23:00:00.236722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:00.236692 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" Oct 29 23:00:00.271175 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:00.271137 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5"] Oct 29 23:00:00.287287 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:00.287256 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:00:00.323897 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:00.323851 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2txr\" (UniqueName: \"kubernetes.io/projected/146f0c5f-f2c8-458c-958f-72cf0d864c92-kube-api-access-w2txr\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5\" (UID: \"146f0c5f-f2c8-458c-958f-72cf0d864c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" Oct 29 23:00:00.324145 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:00.324110 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/146f0c5f-f2c8-458c-958f-72cf0d864c92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5\" (UID: \"146f0c5f-f2c8-458c-958f-72cf0d864c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" Oct 29 23:00:00.425854 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:00.425819 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/146f0c5f-f2c8-458c-958f-72cf0d864c92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5\" (UID: \"146f0c5f-f2c8-458c-958f-72cf0d864c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" Oct 29 23:00:00.426031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:00.425989 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2txr\" (UniqueName: \"kubernetes.io/projected/146f0c5f-f2c8-458c-958f-72cf0d864c92-kube-api-access-w2txr\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5\" (UID: \"146f0c5f-f2c8-458c-958f-72cf0d864c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" Oct 29 23:00:00.426790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:00.426767 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/146f0c5f-f2c8-458c-958f-72cf0d864c92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5\" (UID: \"146f0c5f-f2c8-458c-958f-72cf0d864c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" Oct 29 23:00:00.452001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:00.451968 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2txr\" (UniqueName: \"kubernetes.io/projected/146f0c5f-f2c8-458c-958f-72cf0d864c92-kube-api-access-w2txr\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5\" (UID: \"146f0c5f-f2c8-458c-958f-72cf0d864c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" Oct 29 23:00:00.562940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:00.562845 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" Oct 29 23:00:00.890293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:00.884672 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5"] Oct 29 23:00:01.711480 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:01.711449 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c" Oct 29 23:00:01.842136 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:01.842087 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" event={"ID":"146f0c5f-f2c8-458c-958f-72cf0d864c92","Type":"ContainerStarted","Data":"ff7f70f3e6a5fb9bddb1b4670da68657252dca6840537cb9695a81cd79bc497d"} Oct 29 23:00:01.842558 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:01.842532 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" event={"ID":"146f0c5f-f2c8-458c-958f-72cf0d864c92","Type":"ContainerStarted","Data":"dd68e74293e575a50248d0db24c67a02cdc21d9545b64621dfe434793f0e2075"} Oct 29 23:00:01.912106 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:01.912049 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" podStartSLOduration=1.912030624 podStartE2EDuration="1.912030624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:01.898667694 +0000 UTC m=+1048.550166228" watchObservedRunningTime="2026-10-29 23:00:01.912030624 +0000 UTC m=+1048.563529154" Oct 29 23:00:02.854420 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:02.854381 5057 generic.go:356] "Generic (PLEG): container finished" podID="146f0c5f-f2c8-458c-958f-72cf0d864c92" containerID="ff7f70f3e6a5fb9bddb1b4670da68657252dca6840537cb9695a81cd79bc497d" exitCode=60 Oct 29 23:00:02.854630 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:02.854468 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" event={"ID":"146f0c5f-f2c8-458c-958f-72cf0d864c92","Type":"ContainerDied","Data":"ff7f70f3e6a5fb9bddb1b4670da68657252dca6840537cb9695a81cd79bc497d"} Oct 29 23:00:02.854864 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:02.854846 5057 scope.go:122] "RemoveContainer" containerID="ff7f70f3e6a5fb9bddb1b4670da68657252dca6840537cb9695a81cd79bc497d" Oct 29 23:00:03.414211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:03.410331 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2" podUID="0793f543-6d14-4794-934a-26ea2d40e730" containerName="payload-processing" containerID="cri-o://7c238644441ecd6ec079329487737109811ef5f1829fac3a335f3f28cebc4e7d" gracePeriod=25 Oct 29 23:00:03.867017 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:03.866979 5057 generic.go:356] "Generic (PLEG): container finished" podID="146f0c5f-f2c8-458c-958f-72cf0d864c92" containerID="5c6b220c0f43f99e154451b3ee1ae058516a175b25f7845bcf8b8339fc78195f" exitCode=60 Oct 29 23:00:03.867176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:03.867021 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" event={"ID":"146f0c5f-f2c8-458c-958f-72cf0d864c92","Type":"ContainerDied","Data":"5c6b220c0f43f99e154451b3ee1ae058516a175b25f7845bcf8b8339fc78195f"} Oct 29 23:00:03.867176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:03.867161 5057 scope.go:122] "RemoveContainer" containerID="ff7f70f3e6a5fb9bddb1b4670da68657252dca6840537cb9695a81cd79bc497d" Oct 29 23:00:03.867617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:03.867596 5057 scope.go:122] "RemoveContainer" containerID="5c6b220c0f43f99e154451b3ee1ae058516a175b25f7845bcf8b8339fc78195f" Oct 29 23:00:03.868026 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:00:03.868006 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5_odh-ai-gateway-infra(146f0c5f-f2c8-458c-958f-72cf0d864c92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" podUID="146f0c5f-f2c8-458c-958f-72cf0d864c92" Oct 29 23:00:03.917585 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:00:03.917547 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:00:04.995221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:04.982099 5057 scope.go:122] "RemoveContainer" containerID="5c6b220c0f43f99e154451b3ee1ae058516a175b25f7845bcf8b8339fc78195f" Oct 29 23:00:05.001224 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:00:04.997426 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5_odh-ai-gateway-infra(146f0c5f-f2c8-458c-958f-72cf0d864c92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" podUID="146f0c5f-f2c8-458c-958f-72cf0d864c92" Oct 29 23:00:09.222407 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:09.222361 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2" podUID="0793f543-6d14-4794-934a-26ea2d40e730" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:00:14.473808 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:14.473781 5057 scope.go:122] "RemoveContainer" containerID="5c6b220c0f43f99e154451b3ee1ae058516a175b25f7845bcf8b8339fc78195f" Oct 29 23:00:15.150172 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:15.150090 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" event={"ID":"146f0c5f-f2c8-458c-958f-72cf0d864c92","Type":"ContainerStarted","Data":"631d57d9512042b1cca63763fc93e7b5367221ac0ab66157e2bd8fc5fba3ba98"} Oct 29 23:00:16.161488 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:16.161457 5057 generic.go:356] "Generic (PLEG): container finished" podID="146f0c5f-f2c8-458c-958f-72cf0d864c92" containerID="631d57d9512042b1cca63763fc93e7b5367221ac0ab66157e2bd8fc5fba3ba98" exitCode=60 Oct 29 23:00:16.161936 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:16.161497 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" event={"ID":"146f0c5f-f2c8-458c-958f-72cf0d864c92","Type":"ContainerDied","Data":"631d57d9512042b1cca63763fc93e7b5367221ac0ab66157e2bd8fc5fba3ba98"} Oct 29 23:00:16.161936 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:16.161556 5057 scope.go:122] "RemoveContainer" containerID="5c6b220c0f43f99e154451b3ee1ae058516a175b25f7845bcf8b8339fc78195f" Oct 29 23:00:16.162027 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:16.161995 5057 scope.go:122] "RemoveContainer" containerID="631d57d9512042b1cca63763fc93e7b5367221ac0ab66157e2bd8fc5fba3ba98" Oct 29 23:00:16.162462 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:00:16.162437 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5_odh-ai-gateway-infra(146f0c5f-f2c8-458c-958f-72cf0d864c92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" podUID="146f0c5f-f2c8-458c-958f-72cf0d864c92" Oct 29 23:00:16.205314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:16.205282 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5"] Oct 29 23:00:17.776997 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:17.776964 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" Oct 29 23:00:17.841038 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:17.840952 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/146f0c5f-f2c8-458c-958f-72cf0d864c92-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/146f0c5f-f2c8-458c-958f-72cf0d864c92-maas-api-service-ca\") pod \"146f0c5f-f2c8-458c-958f-72cf0d864c92\" (UID: \"146f0c5f-f2c8-458c-958f-72cf0d864c92\") " Oct 29 23:00:17.841252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:17.841227 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146f0c5f-f2c8-458c-958f-72cf0d864c92-kube-api-access-w2txr\" (UniqueName: \"kubernetes.io/projected/146f0c5f-f2c8-458c-958f-72cf0d864c92-kube-api-access-w2txr\") pod \"146f0c5f-f2c8-458c-958f-72cf0d864c92\" (UID: \"146f0c5f-f2c8-458c-958f-72cf0d864c92\") " Oct 29 23:00:17.841449 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:17.841420 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146f0c5f-f2c8-458c-958f-72cf0d864c92-maas-api-service-ca" pod "146f0c5f-f2c8-458c-958f-72cf0d864c92" (UID: "146f0c5f-f2c8-458c-958f-72cf0d864c92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:17.842020 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:17.841998 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/146f0c5f-f2c8-458c-958f-72cf0d864c92-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:17.844010 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:17.843977 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146f0c5f-f2c8-458c-958f-72cf0d864c92-kube-api-access-w2txr" pod "146f0c5f-f2c8-458c-958f-72cf0d864c92" (UID: "146f0c5f-f2c8-458c-958f-72cf0d864c92"). InnerVolumeSpecName "kube-api-access-w2txr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:17.945969 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:17.945930 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2txr\" (UniqueName: \"kubernetes.io/projected/146f0c5f-f2c8-458c-958f-72cf0d864c92-kube-api-access-w2txr\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:18.230754 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:18.230716 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" event={"ID":"146f0c5f-f2c8-458c-958f-72cf0d864c92","Type":"ContainerDied","Data":"dd68e74293e575a50248d0db24c67a02cdc21d9545b64621dfe434793f0e2075"} Oct 29 23:00:18.231154 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:18.231115 5057 scope.go:122] "RemoveContainer" containerID="631d57d9512042b1cca63763fc93e7b5367221ac0ab66157e2bd8fc5fba3ba98" Oct 29 23:00:18.231393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:18.231041 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5" Oct 29 23:00:18.369589 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:18.369557 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5"] Oct 29 23:00:18.376479 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:18.376448 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888580-rvgd5"] Oct 29 23:00:19.215385 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:19.215348 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2" podUID="0793f543-6d14-4794-934a-26ea2d40e730" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:00:19.484883 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:19.484803 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="146f0c5f-f2c8-458c-958f-72cf0d864c92" path="/var/lib/kubelet/pods/146f0c5f-f2c8-458c-958f-72cf0d864c92/volumes" Oct 29 23:00:21.351209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:21.342595 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w"] Oct 29 23:00:23.761636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:23.761603 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2" Oct 29 23:00:23.782706 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:23.782677 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0793f543-6d14-4794-934a-26ea2d40e730-kube-api-access-56xgd\" (UniqueName: \"kubernetes.io/projected/0793f543-6d14-4794-934a-26ea2d40e730-kube-api-access-56xgd\") pod \"0793f543-6d14-4794-934a-26ea2d40e730\" (UID: \"0793f543-6d14-4794-934a-26ea2d40e730\") " Oct 29 23:00:23.782858 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:23.782757 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0793f543-6d14-4794-934a-26ea2d40e730-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0793f543-6d14-4794-934a-26ea2d40e730-plugins-config-volume\") pod \"0793f543-6d14-4794-934a-26ea2d40e730\" (UID: \"0793f543-6d14-4794-934a-26ea2d40e730\") " Oct 29 23:00:23.783214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:23.783172 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0793f543-6d14-4794-934a-26ea2d40e730-plugins-config-volume" pod "0793f543-6d14-4794-934a-26ea2d40e730" (UID: "0793f543-6d14-4794-934a-26ea2d40e730"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:23.783484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:23.783465 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0793f543-6d14-4794-934a-26ea2d40e730-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:23.784965 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:23.784933 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0793f543-6d14-4794-934a-26ea2d40e730-kube-api-access-56xgd" pod "0793f543-6d14-4794-934a-26ea2d40e730" (UID: "0793f543-6d14-4794-934a-26ea2d40e730"). InnerVolumeSpecName "kube-api-access-56xgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:23.884829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:23.884796 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56xgd\" (UniqueName: \"kubernetes.io/projected/0793f543-6d14-4794-934a-26ea2d40e730-kube-api-access-56xgd\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:24.327045 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:24.327007 5057 generic.go:356] "Generic (PLEG): container finished" podID="0793f543-6d14-4794-934a-26ea2d40e730" containerID="7c238644441ecd6ec079329487737109811ef5f1829fac3a335f3f28cebc4e7d" exitCode=0 Oct 29 23:00:24.327267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:24.327051 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2" event={"ID":"0793f543-6d14-4794-934a-26ea2d40e730","Type":"ContainerDied","Data":"7c238644441ecd6ec079329487737109811ef5f1829fac3a335f3f28cebc4e7d"} Oct 29 23:00:24.327267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:24.327067 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2" Oct 29 23:00:24.327267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:24.327137 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2" event={"ID":"0793f543-6d14-4794-934a-26ea2d40e730","Type":"ContainerDied","Data":"962df39aed95e3ba812028ba1beca1526c1b1108ae2b5524a59e4069c9b4f86f"} Oct 29 23:00:24.327267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:24.327167 5057 scope.go:122] "RemoveContainer" containerID="7c238644441ecd6ec079329487737109811ef5f1829fac3a335f3f28cebc4e7d" Oct 29 23:00:24.338634 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:24.338601 5057 scope.go:122] "RemoveContainer" containerID="7c238644441ecd6ec079329487737109811ef5f1829fac3a335f3f28cebc4e7d" Oct 29 23:00:24.338946 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:00:24.338924 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c238644441ecd6ec079329487737109811ef5f1829fac3a335f3f28cebc4e7d\": container with ID starting with 7c238644441ecd6ec079329487737109811ef5f1829fac3a335f3f28cebc4e7d not found: ID does not exist" containerID="7c238644441ecd6ec079329487737109811ef5f1829fac3a335f3f28cebc4e7d" Oct 29 23:00:24.339040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:24.338954 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c238644441ecd6ec079329487737109811ef5f1829fac3a335f3f28cebc4e7d"} err="failed to get container status \"7c238644441ecd6ec079329487737109811ef5f1829fac3a335f3f28cebc4e7d\": rpc error: code = NotFound desc = could not find container \"7c238644441ecd6ec079329487737109811ef5f1829fac3a335f3f28cebc4e7d\": container with ID starting with 7c238644441ecd6ec079329487737109811ef5f1829fac3a335f3f28cebc4e7d not found: ID does not exist" Oct 29 23:00:24.524914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:24.524849 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2"] Oct 29 23:00:24.530767 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:24.530740 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-txzw2"] Oct 29 23:00:25.483259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:25.483224 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0793f543-6d14-4794-934a-26ea2d40e730" path="/var/lib/kubelet/pods/0793f543-6d14-4794-934a-26ea2d40e730/volumes" Oct 29 23:00:26.343309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:26.343259 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w" podUID="ff8fe269-cf2b-45ae-8965-640b6404a618" containerName="payload-processing" containerID="cri-o://d9e0f42318f8a44457cbb000c26322eea2c5881974cf6e5b1f05d3e0a6dc5111" gracePeriod=25 Oct 29 23:00:26.675148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:26.675117 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w" Oct 29 23:00:26.745602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:26.745565 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff8fe269-cf2b-45ae-8965-640b6404a618-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8fe269-cf2b-45ae-8965-640b6404a618-plugins-config-volume\") pod \"ff8fe269-cf2b-45ae-8965-640b6404a618\" (UID: \"ff8fe269-cf2b-45ae-8965-640b6404a618\") " Oct 29 23:00:26.746074 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:26.746046 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff8fe269-cf2b-45ae-8965-640b6404a618-kube-api-access-frbgm\" (UniqueName: \"kubernetes.io/projected/ff8fe269-cf2b-45ae-8965-640b6404a618-kube-api-access-frbgm\") pod \"ff8fe269-cf2b-45ae-8965-640b6404a618\" (UID: \"ff8fe269-cf2b-45ae-8965-640b6404a618\") " Oct 29 23:00:26.746697 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:26.746611 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff8fe269-cf2b-45ae-8965-640b6404a618-plugins-config-volume" pod "ff8fe269-cf2b-45ae-8965-640b6404a618" (UID: "ff8fe269-cf2b-45ae-8965-640b6404a618"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:26.757245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:26.757215 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8fe269-cf2b-45ae-8965-640b6404a618-kube-api-access-frbgm" pod "ff8fe269-cf2b-45ae-8965-640b6404a618" (UID: "ff8fe269-cf2b-45ae-8965-640b6404a618"). InnerVolumeSpecName "kube-api-access-frbgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:26.848399 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:26.848294 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frbgm\" (UniqueName: \"kubernetes.io/projected/ff8fe269-cf2b-45ae-8965-640b6404a618-kube-api-access-frbgm\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:26.848399 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:26.848332 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8fe269-cf2b-45ae-8965-640b6404a618-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:27.356664 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:27.356626 5057 generic.go:356] "Generic (PLEG): container finished" podID="ff8fe269-cf2b-45ae-8965-640b6404a618" containerID="d9e0f42318f8a44457cbb000c26322eea2c5881974cf6e5b1f05d3e0a6dc5111" exitCode=0 Oct 29 23:00:27.356873 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:27.356674 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w" Oct 29 23:00:27.356873 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:27.356719 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w" event={"ID":"ff8fe269-cf2b-45ae-8965-640b6404a618","Type":"ContainerDied","Data":"d9e0f42318f8a44457cbb000c26322eea2c5881974cf6e5b1f05d3e0a6dc5111"} Oct 29 23:00:27.356873 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:27.356774 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w" event={"ID":"ff8fe269-cf2b-45ae-8965-640b6404a618","Type":"ContainerDied","Data":"66db97bce45b9c1a08a2e07afd6ea4ed00ec0f7ee4bb7809896681f8331f1203"} Oct 29 23:00:27.356873 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:27.356804 5057 scope.go:122] "RemoveContainer" containerID="d9e0f42318f8a44457cbb000c26322eea2c5881974cf6e5b1f05d3e0a6dc5111" Oct 29 23:00:27.380587 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:27.380564 5057 scope.go:122] "RemoveContainer" containerID="d9e0f42318f8a44457cbb000c26322eea2c5881974cf6e5b1f05d3e0a6dc5111" Oct 29 23:00:27.380878 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:00:27.380857 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9e0f42318f8a44457cbb000c26322eea2c5881974cf6e5b1f05d3e0a6dc5111\": container with ID starting with d9e0f42318f8a44457cbb000c26322eea2c5881974cf6e5b1f05d3e0a6dc5111 not found: ID does not exist" containerID="d9e0f42318f8a44457cbb000c26322eea2c5881974cf6e5b1f05d3e0a6dc5111" Oct 29 23:00:27.380926 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:27.380885 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9e0f42318f8a44457cbb000c26322eea2c5881974cf6e5b1f05d3e0a6dc5111"} err="failed to get container status \"d9e0f42318f8a44457cbb000c26322eea2c5881974cf6e5b1f05d3e0a6dc5111\": rpc error: code = NotFound desc = could not find container \"d9e0f42318f8a44457cbb000c26322eea2c5881974cf6e5b1f05d3e0a6dc5111\": container with ID starting with d9e0f42318f8a44457cbb000c26322eea2c5881974cf6e5b1f05d3e0a6dc5111 not found: ID does not exist" Oct 29 23:00:27.418340 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:27.418312 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w"] Oct 29 23:00:27.434562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:27.434525 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-660540-5894d7977d-xlh2w"] Oct 29 23:00:27.483606 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:27.483573 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff8fe269-cf2b-45ae-8965-640b6404a618" path="/var/lib/kubelet/pods/ff8fe269-cf2b-45ae-8965-640b6404a618/volumes" Oct 29 23:00:28.797949 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:28.797909 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s"] Oct 29 23:00:28.798451 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:28.798156 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" podUID="4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7" containerName="maas-api" containerID="cri-o://b6a2af0a6de95c74fc6c85b20c2eb2599a7743b9d62486a3ad3cfed25b58fe97" gracePeriod=30 Oct 29 23:00:29.629332 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:29.628002 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" Oct 29 23:00:29.643308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:29.641891 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-maas-api-tls\") pod \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\" (UID: \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\") " Oct 29 23:00:29.643308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:29.642068 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-kube-api-access-c5h9x\" (UniqueName: \"kubernetes.io/projected/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-kube-api-access-c5h9x\") pod \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\" (UID: \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\") " Oct 29 23:00:29.643308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:29.642155 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-metrics-certs\") pod \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\" (UID: \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\") " Oct 29 23:00:29.643308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:29.642243 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-openshift-service-ca\") pod \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\" (UID: \"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7\") " Oct 29 23:00:29.645509 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:29.645480 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-openshift-service-ca" pod "4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7" (UID: "4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:29.666030 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:29.665997 5057 generic.go:356] "Generic (PLEG): container finished" podID="4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7" containerID="b6a2af0a6de95c74fc6c85b20c2eb2599a7743b9d62486a3ad3cfed25b58fe97" exitCode=0 Oct 29 23:00:29.666643 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:29.666610 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" Oct 29 23:00:29.747795 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:29.747764 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:29.754348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:29.754281 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-maas-api-tls" pod "4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7" (UID: "4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:29.763676 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:29.763632 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" event={"ID":"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7","Type":"ContainerDied","Data":"b6a2af0a6de95c74fc6c85b20c2eb2599a7743b9d62486a3ad3cfed25b58fe97"} Oct 29 23:00:29.763956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:29.763916 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s" event={"ID":"4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7","Type":"ContainerDied","Data":"cbb24be29097bba9e85b4f5837fd358acff89d373ca5557a3a0917fa35e3c2c1"} Oct 29 23:00:29.764693 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:29.764083 5057 scope.go:122] "RemoveContainer" containerID="b6a2af0a6de95c74fc6c85b20c2eb2599a7743b9d62486a3ad3cfed25b58fe97" Oct 29 23:00:29.816823 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:29.816767 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-kube-api-access-c5h9x" pod "4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7" (UID: "4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7"). InnerVolumeSpecName "kube-api-access-c5h9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:29.817739 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:29.817694 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-metrics-certs" pod "4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7" (UID: "4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:29.846066 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:29.845742 5057 scope.go:122] "RemoveContainer" containerID="b6a2af0a6de95c74fc6c85b20c2eb2599a7743b9d62486a3ad3cfed25b58fe97" Oct 29 23:00:29.847839 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:00:29.847716 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a2af0a6de95c74fc6c85b20c2eb2599a7743b9d62486a3ad3cfed25b58fe97\": container with ID starting with b6a2af0a6de95c74fc6c85b20c2eb2599a7743b9d62486a3ad3cfed25b58fe97 not found: ID does not exist" containerID="b6a2af0a6de95c74fc6c85b20c2eb2599a7743b9d62486a3ad3cfed25b58fe97" Oct 29 23:00:29.847839 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:29.847766 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a2af0a6de95c74fc6c85b20c2eb2599a7743b9d62486a3ad3cfed25b58fe97"} err="failed to get container status \"b6a2af0a6de95c74fc6c85b20c2eb2599a7743b9d62486a3ad3cfed25b58fe97\": rpc error: code = NotFound desc = could not find container \"b6a2af0a6de95c74fc6c85b20c2eb2599a7743b9d62486a3ad3cfed25b58fe97\": container with ID starting with b6a2af0a6de95c74fc6c85b20c2eb2599a7743b9d62486a3ad3cfed25b58fe97 not found: ID does not exist" Oct 29 23:00:29.850497 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:29.850471 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:29.850667 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:29.850650 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5h9x\" (UniqueName: \"kubernetes.io/projected/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-kube-api-access-c5h9x\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:29.850807 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:29.850776 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:30.137803 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:30.137735 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s"] Oct 29 23:00:30.150299 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:30.150264 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c79a4e-55fcb94688-qvn4s"] Oct 29 23:00:31.483182 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:31.483145 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7" path="/var/lib/kubelet/pods/4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7/volumes" Oct 29 23:00:33.368515 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:33.368479 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr"] Oct 29 23:00:33.368904 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:33.368743 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" podUID="7723f51d-300b-486d-9ce1-e69a034e0939" containerName="maas-api" containerID="cri-o://89ce811176c5fa41dab630fa9f169c0e69e2455df156a928086192041454890d" gracePeriod=30 Oct 29 23:00:33.967483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:33.967448 5057 generic.go:356] "Generic (PLEG): container finished" podID="7723f51d-300b-486d-9ce1-e69a034e0939" containerID="89ce811176c5fa41dab630fa9f169c0e69e2455df156a928086192041454890d" exitCode=0 Oct 29 23:00:33.967800 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:33.967585 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" event={"ID":"7723f51d-300b-486d-9ce1-e69a034e0939","Type":"ContainerDied","Data":"89ce811176c5fa41dab630fa9f169c0e69e2455df156a928086192041454890d"} Oct 29 23:00:34.159213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:34.124315 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" Oct 29 23:00:34.159213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:34.155967 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7723f51d-300b-486d-9ce1-e69a034e0939-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7723f51d-300b-486d-9ce1-e69a034e0939-openshift-service-ca\") pod \"7723f51d-300b-486d-9ce1-e69a034e0939\" (UID: \"7723f51d-300b-486d-9ce1-e69a034e0939\") " Oct 29 23:00:34.159213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:34.156108 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7723f51d-300b-486d-9ce1-e69a034e0939-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7723f51d-300b-486d-9ce1-e69a034e0939-metrics-certs\") pod \"7723f51d-300b-486d-9ce1-e69a034e0939\" (UID: \"7723f51d-300b-486d-9ce1-e69a034e0939\") " Oct 29 23:00:34.159213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:34.156176 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7723f51d-300b-486d-9ce1-e69a034e0939-kube-api-access-kplcr\" (UniqueName: \"kubernetes.io/projected/7723f51d-300b-486d-9ce1-e69a034e0939-kube-api-access-kplcr\") pod \"7723f51d-300b-486d-9ce1-e69a034e0939\" (UID: \"7723f51d-300b-486d-9ce1-e69a034e0939\") " Oct 29 23:00:34.159213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:34.156250 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7723f51d-300b-486d-9ce1-e69a034e0939-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7723f51d-300b-486d-9ce1-e69a034e0939-maas-api-tls\") pod \"7723f51d-300b-486d-9ce1-e69a034e0939\" (UID: \"7723f51d-300b-486d-9ce1-e69a034e0939\") " Oct 29 23:00:34.177473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:34.177435 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7723f51d-300b-486d-9ce1-e69a034e0939-openshift-service-ca" pod "7723f51d-300b-486d-9ce1-e69a034e0939" (UID: "7723f51d-300b-486d-9ce1-e69a034e0939"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:34.193531 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:00:34.193492 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:00:34.320482 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:34.317859 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7723f51d-300b-486d-9ce1-e69a034e0939-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:34.405908 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:34.368316 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7723f51d-300b-486d-9ce1-e69a034e0939-metrics-certs" pod "7723f51d-300b-486d-9ce1-e69a034e0939" (UID: "7723f51d-300b-486d-9ce1-e69a034e0939"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:34.419557 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:34.419515 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7723f51d-300b-486d-9ce1-e69a034e0939-maas-api-tls" pod "7723f51d-300b-486d-9ce1-e69a034e0939" (UID: "7723f51d-300b-486d-9ce1-e69a034e0939"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:34.439154 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:34.439118 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7723f51d-300b-486d-9ce1-e69a034e0939-kube-api-access-kplcr" pod "7723f51d-300b-486d-9ce1-e69a034e0939" (UID: "7723f51d-300b-486d-9ce1-e69a034e0939"). InnerVolumeSpecName "kube-api-access-kplcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:34.460910 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:34.460284 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7723f51d-300b-486d-9ce1-e69a034e0939-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:34.460910 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:34.460330 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kplcr\" (UniqueName: \"kubernetes.io/projected/7723f51d-300b-486d-9ce1-e69a034e0939-kube-api-access-kplcr\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:34.460910 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:34.460348 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7723f51d-300b-486d-9ce1-e69a034e0939-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:35.029215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.023455 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" event={"ID":"7723f51d-300b-486d-9ce1-e69a034e0939","Type":"ContainerDied","Data":"7d6e575fa0f81162dc6bd8a69fba8adf225f25eb861367be932d831349da8c0b"} Oct 29 23:00:35.029215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.023551 5057 scope.go:122] "RemoveContainer" containerID="89ce811176c5fa41dab630fa9f169c0e69e2455df156a928086192041454890d" Oct 29 23:00:35.029215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.023731 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" Oct 29 23:00:35.123298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.123261 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr"] Oct 29 23:00:35.136548 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.136516 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr"] Oct 29 23:00:35.194460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.192435 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd"] Oct 29 23:00:35.194460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.193895 5057 cpu_manager.go:414] "containerMap: removing container" podUID="146f0c5f-f2c8-458c-958f-72cf0d864c92" containerName="cleanup" Oct 29 23:00:35.194460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.193917 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146f0c5f-f2c8-458c-958f-72cf0d864c92" containerName="cleanup" Oct 29 23:00:35.194460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.193936 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7723f51d-300b-486d-9ce1-e69a034e0939" containerName="maas-api" Oct 29 23:00:35.194460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.193944 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7723f51d-300b-486d-9ce1-e69a034e0939" containerName="maas-api" Oct 29 23:00:35.194460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.193958 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7" containerName="maas-api" Oct 29 23:00:35.194460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.193965 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7" containerName="maas-api" Oct 29 23:00:35.194460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.193977 5057 cpu_manager.go:414] "containerMap: removing container" podUID="ff8fe269-cf2b-45ae-8965-640b6404a618" containerName="payload-processing" Oct 29 23:00:35.194460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.193984 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff8fe269-cf2b-45ae-8965-640b6404a618" containerName="payload-processing" Oct 29 23:00:35.194460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.194002 5057 cpu_manager.go:414] "containerMap: removing container" podUID="146f0c5f-f2c8-458c-958f-72cf0d864c92" containerName="cleanup" Oct 29 23:00:35.194460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.194011 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146f0c5f-f2c8-458c-958f-72cf0d864c92" containerName="cleanup" Oct 29 23:00:35.194460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.194068 5057 cpu_manager.go:414] "containerMap: removing container" podUID="0793f543-6d14-4794-934a-26ea2d40e730" containerName="payload-processing" Oct 29 23:00:35.194460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.194075 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0793f543-6d14-4794-934a-26ea2d40e730" containerName="payload-processing" Oct 29 23:00:35.194460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.194259 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="0793f543-6d14-4794-934a-26ea2d40e730" containerName="payload-processing" Oct 29 23:00:35.194460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.194278 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="146f0c5f-f2c8-458c-958f-72cf0d864c92" containerName="cleanup" Oct 29 23:00:35.194460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.194288 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7723f51d-300b-486d-9ce1-e69a034e0939" containerName="maas-api" Oct 29 23:00:35.194460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.194299 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6a3c57-482b-4f5a-a1fb-a7f09a9af6e7" containerName="maas-api" Oct 29 23:00:35.194460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.194309 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff8fe269-cf2b-45ae-8965-640b6404a618" containerName="payload-processing" Oct 29 23:00:35.194460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.194327 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="146f0c5f-f2c8-458c-958f-72cf0d864c92" containerName="cleanup" Oct 29 23:00:35.201639 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.201458 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.221015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.220977 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd"] Oct 29 23:00:35.224947 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.224924 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ec4f6d-openshift-default-dockercfg-rrkm9\"" Oct 29 23:00:35.304267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.304090 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9864c587-073a-4b1e-9e7c-68fe058b3c16-istiod-ca-cert\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.304267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.304155 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-token\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.304267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.304238 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-podinfo\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.304569 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.304270 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-credential-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.304569 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.304301 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-envoy\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.304569 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.304329 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-workload-certs\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.304569 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.304498 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-data\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.304569 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.304532 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq66n\" (UniqueName: \"kubernetes.io/projected/9864c587-073a-4b1e-9e7c-68fe058b3c16-kube-api-access-gq66n\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.304836 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.304605 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-workload-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.421223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.406231 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-data\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.421223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.406302 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq66n\" (UniqueName: \"kubernetes.io/projected/9864c587-073a-4b1e-9e7c-68fe058b3c16-kube-api-access-gq66n\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.421223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.406368 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-workload-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.421223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.406580 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9864c587-073a-4b1e-9e7c-68fe058b3c16-istiod-ca-cert\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.421223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.406628 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-token\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.421223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.406686 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-podinfo\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.421223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.406717 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-credential-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.421223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.406741 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-envoy\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.421223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.406770 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-workload-certs\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.423266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.422673 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-workload-certs\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.423266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.422947 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-credential-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.427208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.424499 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-data\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.427208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.425071 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-workload-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.439773 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.439740 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9864c587-073a-4b1e-9e7c-68fe058b3c16-istiod-ca-cert\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.481211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.479529 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-envoy\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.481211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.480025 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-podinfo\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.553177 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.546948 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq66n\" (UniqueName: \"kubernetes.io/projected/9864c587-073a-4b1e-9e7c-68fe058b3c16-kube-api-access-gq66n\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.576863 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.576730 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-token\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:35.578254 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.578211 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7723f51d-300b-486d-9ce1-e69a034e0939" path="/var/lib/kubelet/pods/7723f51d-300b-486d-9ce1-e69a034e0939/volumes" Oct 29 23:00:35.843449 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:35.840716 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:36.179110 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:00:36.179080 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9864c587_073a_4b1e_9e7c_68fe058b3c16.slice/crio-208d919a9e21d16de982e7403d756f05b7ef22f653637e0cf903f879b304ad60 WatchSource:0}: Error finding container 208d919a9e21d16de982e7403d756f05b7ef22f653637e0cf903f879b304ad60: Status 404 returned error can't find the container with id 208d919a9e21d16de982e7403d756f05b7ef22f653637e0cf903f879b304ad60 Oct 29 23:00:36.180085 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:36.180062 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd"] Oct 29 23:00:36.181949 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:36.181918 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:00:36.182023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:36.181986 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:00:36.182023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:36.182013 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:00:36.868046 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:36.867999 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-rntrr" podUID="7723f51d-300b-486d-9ce1-e69a034e0939" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.94:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:37.078580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:37.078539 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" event={"ID":"9864c587-073a-4b1e-9e7c-68fe058b3c16","Type":"ContainerStarted","Data":"9603457875dc61f1e283129cf13a3ff129ec11b88905bc93ac83fe06f50fcf27"} Oct 29 23:00:37.078780 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:37.078603 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" event={"ID":"9864c587-073a-4b1e-9e7c-68fe058b3c16","Type":"ContainerStarted","Data":"208d919a9e21d16de982e7403d756f05b7ef22f653637e0cf903f879b304ad60"} Oct 29 23:00:37.170267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:37.168738 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" podStartSLOduration=2.168716564 podStartE2EDuration="2.168716564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:37.142695088 +0000 UTC m=+1083.794193633" watchObservedRunningTime="2026-10-29 23:00:37.168716564 +0000 UTC m=+1083.820215095" Oct 29 23:00:37.760211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:37.538473 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4"] Oct 29 23:00:37.991220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:37.986150 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" podUID="e3cd3c9e-4e7d-4395-81da-5f43c01467ee" containerName="istio-proxy" containerID="cri-o://243276169f0507fa1085a3a6a2f67d16417494c469f1468227e80f3634580c93" gracePeriod=30 Oct 29 23:00:38.062037 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:38.062005 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:38.062378 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:38.062349 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:38.087101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:38.087064 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:38.088051 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:38.088025 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:00:38.903212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:38.902708 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:00:39.882123 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:39.882090 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx"] Oct 29 23:00:39.882887 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:39.882871 5057 cpu_manager.go:414] "containerMap: removing container" podUID="146f0c5f-f2c8-458c-958f-72cf0d864c92" containerName="cleanup" Oct 29 23:00:39.882887 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:39.882887 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146f0c5f-f2c8-458c-958f-72cf0d864c92" containerName="cleanup" Oct 29 23:00:39.883043 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:39.883034 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="146f0c5f-f2c8-458c-958f-72cf0d864c92" containerName="cleanup" Oct 29 23:00:39.888117 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:39.888097 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" Oct 29 23:00:39.906104 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:39.906079 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b7f715-3c33-483f-9d7c-75797d057663-metrics-certs\") pod \"maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx\" (UID: \"f2b7f715-3c33-483f-9d7c-75797d057663\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" Oct 29 23:00:39.906248 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:39.906209 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b7f715-3c33-483f-9d7c-75797d057663-maas-api-tls\") pod \"maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx\" (UID: \"f2b7f715-3c33-483f-9d7c-75797d057663\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" Oct 29 23:00:39.906399 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:39.906383 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnxkj\" (UniqueName: \"kubernetes.io/projected/f2b7f715-3c33-483f-9d7c-75797d057663-kube-api-access-fnxkj\") pod \"maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx\" (UID: \"f2b7f715-3c33-483f-9d7c-75797d057663\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" Oct 29 23:00:39.906441 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:39.906420 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b7f715-3c33-483f-9d7c-75797d057663-openshift-service-ca\") pod \"maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx\" (UID: \"f2b7f715-3c33-483f-9d7c-75797d057663\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" Oct 29 23:00:39.976298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:39.976262 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-7fa011-serving-cert\"" Oct 29 23:00:39.983463 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:39.983432 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx"] Oct 29 23:00:40.007455 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:40.007418 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnxkj\" (UniqueName: \"kubernetes.io/projected/f2b7f715-3c33-483f-9d7c-75797d057663-kube-api-access-fnxkj\") pod \"maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx\" (UID: \"f2b7f715-3c33-483f-9d7c-75797d057663\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" Oct 29 23:00:40.007638 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:40.007488 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b7f715-3c33-483f-9d7c-75797d057663-openshift-service-ca\") pod \"maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx\" (UID: \"f2b7f715-3c33-483f-9d7c-75797d057663\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" Oct 29 23:00:40.007638 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:40.007614 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b7f715-3c33-483f-9d7c-75797d057663-metrics-certs\") pod \"maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx\" (UID: \"f2b7f715-3c33-483f-9d7c-75797d057663\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" Oct 29 23:00:40.007800 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:40.007676 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b7f715-3c33-483f-9d7c-75797d057663-maas-api-tls\") pod \"maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx\" (UID: \"f2b7f715-3c33-483f-9d7c-75797d057663\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" Oct 29 23:00:40.008284 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:40.008260 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b7f715-3c33-483f-9d7c-75797d057663-openshift-service-ca\") pod \"maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx\" (UID: \"f2b7f715-3c33-483f-9d7c-75797d057663\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" Oct 29 23:00:40.010141 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:40.010118 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b7f715-3c33-483f-9d7c-75797d057663-metrics-certs\") pod \"maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx\" (UID: \"f2b7f715-3c33-483f-9d7c-75797d057663\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" Oct 29 23:00:40.010273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:40.010210 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b7f715-3c33-483f-9d7c-75797d057663-maas-api-tls\") pod \"maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx\" (UID: \"f2b7f715-3c33-483f-9d7c-75797d057663\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" Oct 29 23:00:40.033596 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:40.033563 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnxkj\" (UniqueName: \"kubernetes.io/projected/f2b7f715-3c33-483f-9d7c-75797d057663-kube-api-access-fnxkj\") pod \"maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx\" (UID: \"f2b7f715-3c33-483f-9d7c-75797d057663\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" Oct 29 23:00:40.232481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:40.232173 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" Oct 29 23:00:40.469914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:40.469875 5057 patch_prober.go:29] interesting pod/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:00:40.470576 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:40.470164 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" podUID="e3cd3c9e-4e7d-4395-81da-5f43c01467ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:00:41.684939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:41.678876 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx"] Oct 29 23:00:43.161831 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:00:43.159140 5057 container.go:588] Failed to update stats for container "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3cd3c9e_4e7d_4395_81da_5f43c01467ee.slice/crio-74dfea150b5cdb507c18addc5d661c502d6e62d0fa83b97e5684e19f92b27b57": error while statting cgroup v2: [unable to parse /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3cd3c9e_4e7d_4395_81da_5f43c01467ee.slice/crio-74dfea150b5cdb507c18addc5d661c502d6e62d0fa83b97e5684e19f92b27b57/memory.stat: read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3cd3c9e_4e7d_4395_81da_5f43c01467ee.slice/crio-74dfea150b5cdb507c18addc5d661c502d6e62d0fa83b97e5684e19f92b27b57/memory.stat: no such device], continuing to push stats Oct 29 23:00:43.239235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:43.239093 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7"] Oct 29 23:00:43.292469 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:43.282425 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:43.382456 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:43.357641 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" event={"ID":"f2b7f715-3c33-483f-9d7c-75797d057663","Type":"ContainerStarted","Data":"36813a7f68a19e92ff2bcd4f3a62970df9bcab57642834fe393d1edc7d526792"} Oct 29 23:00:43.382456 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:43.357807 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7" Oct 29 23:00:43.491590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:43.491556 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ec4f6d\"" Oct 29 23:00:43.559209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:43.554532 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ec4f6d-dockercfg-h6xr7\"" Oct 29 23:00:43.628264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:43.614966 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7\" (UID: \"299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7" Oct 29 23:00:43.628264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:43.615105 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhmbg\" (UniqueName: \"kubernetes.io/projected/299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0-kube-api-access-lhmbg\") pod \"payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7\" (UID: \"299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7" Oct 29 23:00:43.628264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:43.620827 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 23:00:43.655154 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:43.639017 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 23:00:43.757263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:43.757146 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7\" (UID: \"299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7" Oct 29 23:00:43.757644 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:43.757592 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhmbg\" (UniqueName: \"kubernetes.io/projected/299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0-kube-api-access-lhmbg\") pod \"payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7\" (UID: \"299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7" Oct 29 23:00:43.784956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:43.784923 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7\" (UID: \"299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7" Oct 29 23:00:44.190242 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:44.161460 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:44.211370 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:44.211241 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhmbg\" (UniqueName: \"kubernetes.io/projected/299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0-kube-api-access-lhmbg\") pod \"payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7\" (UID: \"299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7" Oct 29 23:00:44.581095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:44.581002 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:45.203226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:44.623898 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:45.204054 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:44.886133 5057 generic.go:356] "Generic (PLEG): container finished" podID="e3cd3c9e-4e7d-4395-81da-5f43c01467ee" containerID="243276169f0507fa1085a3a6a2f67d16417494c469f1468227e80f3634580c93" exitCode=0 Oct 29 23:00:45.204336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:44.886164 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" event={"ID":"e3cd3c9e-4e7d-4395-81da-5f43c01467ee","Type":"ContainerDied","Data":"243276169f0507fa1085a3a6a2f67d16417494c469f1468227e80f3634580c93"} Oct 29 23:00:45.204587 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:45.204570 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:00:45.204747 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:45.204732 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7"] Oct 29 23:00:45.204898 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:45.204885 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv"] Oct 29 23:00:45.207403 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:45.207372 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded" Oct 29 23:00:45.666242 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:45.655021 5057 scope.go:122] "RemoveContainer" containerID="0dee6d22d5168608675727d1786f2d4099b04cb69acee06543ee19eecd266452" Oct 29 23:00:45.994582 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:45.987310 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded" Oct 29 23:00:45.994582 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:45.987447 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7ks2l" podUID="bce69276-30ac-47e6-99df-81608424b3dd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:46.196693 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:46.188177 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 23:00:46.196693 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:46.190665 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7" Oct 29 23:00:46.236452 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:46.228407 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:00:46.271645 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:46.269613 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 23:00:46.600438 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:46.600365 5057 trace.go:236] Trace[1067312244]: "Calculate volume metrics of istio-podinfo for pod openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" (29-Oct-2026 23:00:45.237) (total time: 1363ms): Oct 29 23:00:46.600438 ip-10-0-135-80 kubenswrapper[5057]: Trace[1067312244]: [1.36312673s] [1.36312673s] END Oct 29 23:00:46.637338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:46.603770 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" Oct 29 23:00:46.653293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:46.603857 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv" Oct 29 23:00:46.770388 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:46.744435 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 23:00:46.997417 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:46.996224 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97751900-4954-4a2a-bf49-99b702e4a7c2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv\" (UID: \"97751900-4954-4a2a-bf49-99b702e4a7c2\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv" Oct 29 23:00:46.997417 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:46.996339 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l5m9\" (UniqueName: \"kubernetes.io/projected/97751900-4954-4a2a-bf49-99b702e4a7c2-kube-api-access-8l5m9\") pod \"payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv\" (UID: \"97751900-4954-4a2a-bf49-99b702e4a7c2\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv" Oct 29 23:00:47.002417 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.002384 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 23:00:47.216220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.215845 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-data\") pod \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " Oct 29 23:00:47.404059 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.404026 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-envoy\") pod \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " Oct 29 23:00:47.448045 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.447892 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-token\" (UniqueName: \"kubernetes.io/projected/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-token\") pod \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " Oct 29 23:00:47.448665 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.448513 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-credential-socket\") pod \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " Oct 29 23:00:47.449705 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.449684 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-workload-socket\") pod \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " Oct 29 23:00:47.450164 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.450040 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istiod-ca-cert\") pod \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " Oct 29 23:00:47.450489 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.450474 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-workload-certs\") pod \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " Oct 29 23:00:47.797220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.227308 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-data" pod "e3cd3c9e-4e7d-4395-81da-5f43c01467ee" (UID: "e3cd3c9e-4e7d-4395-81da-5f43c01467ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:47.797220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.663262 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-credential-socket" pod "e3cd3c9e-4e7d-4395-81da-5f43c01467ee" (UID: "e3cd3c9e-4e7d-4395-81da-5f43c01467ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:47.797220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.732413 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-workload-certs" pod "e3cd3c9e-4e7d-4395-81da-5f43c01467ee" (UID: "e3cd3c9e-4e7d-4395-81da-5f43c01467ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:47.797220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.236424 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:47.797220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.339445 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:47.811625 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.811586 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-podinfo\") pod \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " Oct 29 23:00:47.813492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.813463 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-kube-api-access-gv8kf\" (UniqueName: \"kubernetes.io/projected/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-kube-api-access-gv8kf\") pod \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " Oct 29 23:00:47.817552 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.817523 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97751900-4954-4a2a-bf49-99b702e4a7c2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv\" (UID: \"97751900-4954-4a2a-bf49-99b702e4a7c2\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv" Oct 29 23:00:47.819820 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.819796 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l5m9\" (UniqueName: \"kubernetes.io/projected/97751900-4954-4a2a-bf49-99b702e4a7c2-kube-api-access-8l5m9\") pod \"payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv\" (UID: \"97751900-4954-4a2a-bf49-99b702e4a7c2\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv" Oct 29 23:00:47.821628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.821602 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:47.821792 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.821775 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:47.821906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.821893 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:47.821994 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.819651 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istiod-ca-cert" pod "e3cd3c9e-4e7d-4395-81da-5f43c01467ee" (UID: "e3cd3c9e-4e7d-4395-81da-5f43c01467ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:47.823067 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.823046 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97751900-4954-4a2a-bf49-99b702e4a7c2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv\" (UID: \"97751900-4954-4a2a-bf49-99b702e4a7c2\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv" Oct 29 23:00:47.828197 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.819266 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:47.875241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:47.226245 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:48.094212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:48.062244 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-token" pod "e3cd3c9e-4e7d-4395-81da-5f43c01467ee" (UID: "e3cd3c9e-4e7d-4395-81da-5f43c01467ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:48.095483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:48.071867 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-workload-socket" pod "e3cd3c9e-4e7d-4395-81da-5f43c01467ee" (UID: "e3cd3c9e-4e7d-4395-81da-5f43c01467ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:48.132251 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:48.097236 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" podStartSLOduration=8.575168172 podStartE2EDuration="9.097215653s" totalImagesPullingTime="522.047481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:45.92870574 +0000 UTC m=+1092.580204287" watchObservedRunningTime="2026-10-29 23:00:48.097215653 +0000 UTC m=+1094.748714187" Oct 29 23:00:48.411791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:48.411751 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:48.412387 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:48.412362 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:48.412514 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:48.412501 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:48.690459 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:48.690405 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-envoy" pod "e3cd3c9e-4e7d-4395-81da-5f43c01467ee" (UID: "e3cd3c9e-4e7d-4395-81da-5f43c01467ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:48.716245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:48.706844 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:48.911673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:48.739889 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l5m9\" (UniqueName: \"kubernetes.io/projected/97751900-4954-4a2a-bf49-99b702e4a7c2-kube-api-access-8l5m9\") pod \"payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv\" (UID: \"97751900-4954-4a2a-bf49-99b702e4a7c2\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv" Oct 29 23:00:49.242365 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:49.242260 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:00:49.277213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:49.248734 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 23:00:49.312328 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:49.312283 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 23:00:49.391219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:49.322053 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:49.401367 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:49.266732 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-podinfo" pod "e3cd3c9e-4e7d-4395-81da-5f43c01467ee" (UID: "e3cd3c9e-4e7d-4395-81da-5f43c01467ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:00:49.401577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:49.338413 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-kube-api-access-gv8kf" pod "e3cd3c9e-4e7d-4395-81da-5f43c01467ee" (UID: "e3cd3c9e-4e7d-4395-81da-5f43c01467ee"). InnerVolumeSpecName "kube-api-access-gv8kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:49.481210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:49.466875 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv" Oct 29 23:00:49.481210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:49.474261 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-podinfo\") pod \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " Oct 29 23:00:49.481210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:49.478281 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-kube-api-access-gv8kf\" (UniqueName: \"kubernetes.io/projected/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-kube-api-access-gv8kf\") pod \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\" (UID: \"e3cd3c9e-4e7d-4395-81da-5f43c01467ee\") " Oct 29 23:00:49.505583 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:49.505495 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" Oct 29 23:00:49.514217 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:00:49.506327 5057 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/e3cd3c9e-4e7d-4395-81da-5f43c01467ee/volumes/kubernetes.io~projected/kube-api-access-gv8kf Oct 29 23:00:49.514551 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:49.514523 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-kube-api-access-gv8kf" pod "e3cd3c9e-4e7d-4395-81da-5f43c01467ee" (UID: "e3cd3c9e-4e7d-4395-81da-5f43c01467ee"). InnerVolumeSpecName "kube-api-access-gv8kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:49.598227 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:49.567518 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv8kf\" (UniqueName: \"kubernetes.io/projected/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-kube-api-access-gv8kf\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:49.678803 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:00:49.572701 5057 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.926s" Oct 29 23:00:49.679114 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:49.679094 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:00:49.681430 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:49.591723 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xf9r8" podUID="63bfe837-48d3-45b6-ae39-e4501036c02a" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.39:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:49.798334 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:00:49.592306 5057 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/e3cd3c9e-4e7d-4395-81da-5f43c01467ee/volumes/kubernetes.io~downward-api/istio-podinfo Oct 29 23:00:49.798334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:49.797974 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-podinfo" pod "e3cd3c9e-4e7d-4395-81da-5f43c01467ee" (UID: "e3cd3c9e-4e7d-4395-81da-5f43c01467ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:00:49.809360 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:49.808026 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3cd3c9e-4e7d-4395-81da-5f43c01467ee-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:49.819419 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:49.819395 5057 trace.go:236] Trace[78923621]: "Calculate volume metrics of istio-envoy for pod openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" (29-Oct-2026 23:00:48.463) (total time: 1356ms): Oct 29 23:00:49.819419 ip-10-0-135-80 kubenswrapper[5057]: Trace[78923621]: [1.356187521s] [1.356187521s] END Oct 29 23:00:50.321488 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.306425 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:00:50.345376 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.345348 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:00:50.345622 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.345604 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:00:50.345771 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.345749 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4" event={"ID":"e3cd3c9e-4e7d-4395-81da-5f43c01467ee","Type":"ContainerDied","Data":"74dfea150b5cdb507c18addc5d661c502d6e62d0fa83b97e5684e19f92b27b57"} Oct 29 23:00:50.355373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.355345 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:00:50.355565 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.355545 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" event={"ID":"f2b7f715-3c33-483f-9d7c-75797d057663","Type":"ContainerStarted","Data":"f7fe21a8787828e18c2397f8a14611827f3ac93282462d103edcdee4f1926625"} Oct 29 23:00:50.355693 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.355677 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv"] Oct 29 23:00:50.355857 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.355842 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:00:50.355979 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.355967 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:00:50.360349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.360325 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" Oct 29 23:00:50.360505 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.360492 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c"] Oct 29 23:00:50.409356 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.360634 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q"] Oct 29 23:00:50.486210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.477529 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e3cd3c9e-4e7d-4395-81da-5f43c01467ee" containerName="istio-proxy" Oct 29 23:00:50.486210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.477564 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3cd3c9e-4e7d-4395-81da-5f43c01467ee" containerName="istio-proxy" Oct 29 23:00:50.486210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.477600 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e3cd3c9e-4e7d-4395-81da-5f43c01467ee" containerName="istio-proxy" Oct 29 23:00:50.486210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.477609 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3cd3c9e-4e7d-4395-81da-5f43c01467ee" containerName="istio-proxy" Oct 29 23:00:50.486210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.477848 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3cd3c9e-4e7d-4395-81da-5f43c01467ee" containerName="istio-proxy" Oct 29 23:00:50.486210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.477866 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3cd3c9e-4e7d-4395-81da-5f43c01467ee" containerName="istio-proxy" Oct 29 23:00:50.495217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.378426 5057 scope.go:122] "RemoveContainer" containerID="243276169f0507fa1085a3a6a2f67d16417494c469f1468227e80f3634580c93" Oct 29 23:00:50.646212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.630476 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q"] Oct 29 23:00:50.646212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.630528 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np"] Oct 29 23:00:50.672365 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.672330 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q" Oct 29 23:00:50.720256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.715624 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-79e6c631-dockercfg-9bjm7\"" Oct 29 23:00:50.720496 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.719949 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjf4m\" (UniqueName: \"kubernetes.io/projected/b774656b-b5c6-4aae-bfa4-e269ced7a278-kube-api-access-sjf4m\") pod \"payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q\" (UID: \"b774656b-b5c6-4aae-bfa4-e269ced7a278\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q" Oct 29 23:00:50.720841 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.720775 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b774656b-b5c6-4aae-bfa4-e269ced7a278-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q\" (UID: \"b774656b-b5c6-4aae-bfa4-e269ced7a278\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q" Oct 29 23:00:50.772212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.742082 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-79e6c631\"" Oct 29 23:00:50.807254 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.807223 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np"] Oct 29 23:00:50.848005 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.832350 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp"] Oct 29 23:00:50.848005 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.832402 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j"] Oct 29 23:00:50.848005 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.832435 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7"] Oct 29 23:00:50.848005 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.809777 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np" Oct 29 23:00:50.876148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.849906 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh9hg\" (UniqueName: \"kubernetes.io/projected/660ff606-64f9-4a49-a739-7d0452742abf-kube-api-access-dh9hg\") pod \"payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np\" (UID: \"660ff606-64f9-4a49-a739-7d0452742abf\") " pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np" Oct 29 23:00:50.876148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.850054 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660ff606-64f9-4a49-a739-7d0452742abf-plugins-config-volume\") pod \"payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np\" (UID: \"660ff606-64f9-4a49-a739-7d0452742abf\") " pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np" Oct 29 23:00:50.876148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.850120 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b774656b-b5c6-4aae-bfa4-e269ced7a278-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q\" (UID: \"b774656b-b5c6-4aae-bfa4-e269ced7a278\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q" Oct 29 23:00:50.876148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.850363 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjf4m\" (UniqueName: \"kubernetes.io/projected/b774656b-b5c6-4aae-bfa4-e269ced7a278-kube-api-access-sjf4m\") pod \"payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q\" (UID: \"b774656b-b5c6-4aae-bfa4-e269ced7a278\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q" Oct 29 23:00:50.901292 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.888953 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b774656b-b5c6-4aae-bfa4-e269ced7a278-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q\" (UID: \"b774656b-b5c6-4aae-bfa4-e269ced7a278\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q" Oct 29 23:00:50.977134 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:00:50.977086 5057 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3cd3c9e_4e7d_4395_81da_5f43c01467ee.slice/crio-74dfea150b5cdb507c18addc5d661c502d6e62d0fa83b97e5684e19f92b27b57\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3cd3c9e_4e7d_4395_81da_5f43c01467ee.slice\": RecentStats: unable to find data in memory cache]" Oct 29 23:00:50.980914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.980859 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh9hg\" (UniqueName: \"kubernetes.io/projected/660ff606-64f9-4a49-a739-7d0452742abf-kube-api-access-dh9hg\") pod \"payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np\" (UID: \"660ff606-64f9-4a49-a739-7d0452742abf\") " pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np" Oct 29 23:00:50.981274 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.981239 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660ff606-64f9-4a49-a739-7d0452742abf-plugins-config-volume\") pod \"payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np\" (UID: \"660ff606-64f9-4a49-a739-7d0452742abf\") " pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np" Oct 29 23:00:50.992209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:50.985304 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660ff606-64f9-4a49-a739-7d0452742abf-plugins-config-volume\") pod \"payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np\" (UID: \"660ff606-64f9-4a49-a739-7d0452742abf\") " pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np" Oct 29 23:00:51.053215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:51.035671 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjf4m\" (UniqueName: \"kubernetes.io/projected/b774656b-b5c6-4aae-bfa4-e269ced7a278-kube-api-access-sjf4m\") pod \"payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q\" (UID: \"b774656b-b5c6-4aae-bfa4-e269ced7a278\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q" Oct 29 23:00:51.106169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:51.106133 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4"] Oct 29 23:00:51.109943 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:51.109915 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh9hg\" (UniqueName: \"kubernetes.io/projected/660ff606-64f9-4a49-a739-7d0452742abf-kube-api-access-dh9hg\") pod \"payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np\" (UID: \"660ff606-64f9-4a49-a739-7d0452742abf\") " pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np" Oct 29 23:00:51.115105 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:51.115071 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7" event={"ID":"299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0","Type":"ContainerStarted","Data":"ca668f412c9696425120496cd09618febd2bc57e3489047b2703f8a4d8a88ecc"} Oct 29 23:00:51.115385 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:51.115368 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c79a4e-openshift-default-f46ddd659-ql8g4"] Oct 29 23:00:51.116254 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:51.116239 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7" Oct 29 23:00:51.217971 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:51.216477 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q" Oct 29 23:00:51.402562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:51.402288 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np" Oct 29 23:00:51.456210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:51.455256 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv"] Oct 29 23:00:51.604718 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:51.604677 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3cd3c9e-4e7d-4395-81da-5f43c01467ee" path="/var/lib/kubelet/pods/e3cd3c9e-4e7d-4395-81da-5f43c01467ee/volumes" Oct 29 23:00:51.779213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:51.769766 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7" podStartSLOduration=8.76974299 podStartE2EDuration="8.76974299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:51.595973429 +0000 UTC m=+1098.247471959" watchObservedRunningTime="2026-10-29 23:00:51.76974299 +0000 UTC m=+1098.421241524" Oct 29 23:00:52.212839 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:52.212126 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7" event={"ID":"299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0","Type":"ContainerStarted","Data":"580ddcff2119fd4c6d6110f20b93a2298260e37832bd5bb6f86373811a77864c"} Oct 29 23:00:52.215136 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:52.215076 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv" event={"ID":"97751900-4954-4a2a-bf49-99b702e4a7c2","Type":"ContainerStarted","Data":"bd1d32b144d1bbb3cfaf404126434bfbce1ed879f9f4001b3c60a813fdddf377"} Oct 29 23:00:52.405473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:52.405086 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np"] Oct 29 23:00:52.485414 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:52.473362 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q"] Oct 29 23:00:53.256261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:53.256176 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np" event={"ID":"660ff606-64f9-4a49-a739-7d0452742abf","Type":"ContainerStarted","Data":"1c368543f61f80c59779028be727a2c9f423b0984f9ed9885052f3a87528e462"} Oct 29 23:00:53.256760 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:53.256277 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np" event={"ID":"660ff606-64f9-4a49-a739-7d0452742abf","Type":"ContainerStarted","Data":"a5efb7bc98ae168deea2059343fa482699d8b2082f761eaf0b58c2f63c3497f1"} Oct 29 23:00:53.257402 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:53.257381 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np" Oct 29 23:00:53.259684 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:53.259557 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q" event={"ID":"b774656b-b5c6-4aae-bfa4-e269ced7a278","Type":"ContainerStarted","Data":"5875e6006f3b9fd69d2f19e204c0a45bcaac4d343d195830322f90bd0ba855f3"} Oct 29 23:00:53.259684 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:53.259622 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q" event={"ID":"b774656b-b5c6-4aae-bfa4-e269ced7a278","Type":"ContainerStarted","Data":"d82c3ad427dfb87ee1247e6c36f39a154b526a50a8a6e7e10c7af7a2efea0337"} Oct 29 23:00:53.259684 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:53.259653 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q" Oct 29 23:00:53.261783 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:53.261757 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv" event={"ID":"97751900-4954-4a2a-bf49-99b702e4a7c2","Type":"ContainerStarted","Data":"bf6440b779b3ba16012554fd4220d8bee29bbf91b41a36aea6798366c974f74e"} Oct 29 23:00:53.261917 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:53.261830 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv" Oct 29 23:00:53.303232 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:53.303163 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np" podStartSLOduration=6.303146805 podStartE2EDuration="6.303146805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:53.300793583 +0000 UTC m=+1099.952292113" watchObservedRunningTime="2026-10-29 23:00:53.303146805 +0000 UTC m=+1099.954645337" Oct 29 23:00:53.358538 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:53.358465 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q" podStartSLOduration=6.358444844 podStartE2EDuration="6.358444844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:53.350952529 +0000 UTC m=+1100.002451059" watchObservedRunningTime="2026-10-29 23:00:53.358444844 +0000 UTC m=+1100.009943375" Oct 29 23:00:53.399337 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:53.399287 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv" podStartSLOduration=10.399273138 podStartE2EDuration="10.399273138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:53.395639584 +0000 UTC m=+1100.047138115" watchObservedRunningTime="2026-10-29 23:00:53.399273138 +0000 UTC m=+1100.050771667" Oct 29 23:00:54.399903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.394250 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm"] Oct 29 23:00:54.526213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.525709 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm"] Oct 29 23:00:54.547398 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.531694 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm" Oct 29 23:00:54.604537 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.549517 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-660540\"" Oct 29 23:00:54.604537 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.549713 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-660540-dockercfg-4s849\"" Oct 29 23:00:54.604537 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.550534 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d"] Oct 29 23:00:54.637376 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.619376 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d" Oct 29 23:00:54.741006 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.737288 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d"] Oct 29 23:00:54.741006 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.740988 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7a76de-9e42-485e-8034-32d15d55834b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d\" (UID: \"ed7a76de-9e42-485e-8034-32d15d55834b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d" Oct 29 23:00:54.741299 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.741034 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7775\" (UniqueName: \"kubernetes.io/projected/7c2efb3f-9de1-42d2-9a4e-ecc08f995e07-kube-api-access-q7775\") pod \"payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm\" (UID: \"7c2efb3f-9de1-42d2-9a4e-ecc08f995e07\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm" Oct 29 23:00:54.741299 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.741206 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2efb3f-9de1-42d2-9a4e-ecc08f995e07-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm\" (UID: \"7c2efb3f-9de1-42d2-9a4e-ecc08f995e07\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm" Oct 29 23:00:54.741299 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.741269 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzxsr\" (UniqueName: \"kubernetes.io/projected/ed7a76de-9e42-485e-8034-32d15d55834b-kube-api-access-rzxsr\") pod \"payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d\" (UID: \"ed7a76de-9e42-485e-8034-32d15d55834b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d" Oct 29 23:00:54.842868 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.842826 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7a76de-9e42-485e-8034-32d15d55834b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d\" (UID: \"ed7a76de-9e42-485e-8034-32d15d55834b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d" Oct 29 23:00:54.844130 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.844104 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7775\" (UniqueName: \"kubernetes.io/projected/7c2efb3f-9de1-42d2-9a4e-ecc08f995e07-kube-api-access-q7775\") pod \"payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm\" (UID: \"7c2efb3f-9de1-42d2-9a4e-ecc08f995e07\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm" Oct 29 23:00:54.844465 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.844443 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2efb3f-9de1-42d2-9a4e-ecc08f995e07-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm\" (UID: \"7c2efb3f-9de1-42d2-9a4e-ecc08f995e07\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm" Oct 29 23:00:54.844648 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.844631 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzxsr\" (UniqueName: \"kubernetes.io/projected/ed7a76de-9e42-485e-8034-32d15d55834b-kube-api-access-rzxsr\") pod \"payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d\" (UID: \"ed7a76de-9e42-485e-8034-32d15d55834b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d" Oct 29 23:00:54.845260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.844050 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7a76de-9e42-485e-8034-32d15d55834b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d\" (UID: \"ed7a76de-9e42-485e-8034-32d15d55834b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d" Oct 29 23:00:54.845428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.845307 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2efb3f-9de1-42d2-9a4e-ecc08f995e07-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm\" (UID: \"7c2efb3f-9de1-42d2-9a4e-ecc08f995e07\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm" Oct 29 23:00:54.870337 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.870302 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzxsr\" (UniqueName: \"kubernetes.io/projected/ed7a76de-9e42-485e-8034-32d15d55834b-kube-api-access-rzxsr\") pod \"payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d\" (UID: \"ed7a76de-9e42-485e-8034-32d15d55834b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d" Oct 29 23:00:54.870764 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.870741 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7775\" (UniqueName: \"kubernetes.io/projected/7c2efb3f-9de1-42d2-9a4e-ecc08f995e07-kube-api-access-q7775\") pod \"payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm\" (UID: \"7c2efb3f-9de1-42d2-9a4e-ecc08f995e07\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm" Oct 29 23:00:54.991493 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:54.991412 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm" Oct 29 23:00:55.039783 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:55.039542 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d" Oct 29 23:00:55.403762 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:55.403432 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c" podUID="e373476e-bc93-4405-b912-ba19c6636462" containerName="payload-pre-processing" containerID="cri-o://8e96911d78fd28730c08e1c3fc31fe2945f761d66970b160ffbff6cd8bca4670" gracePeriod=25 Oct 29 23:00:55.410265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:55.410232 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm"] Oct 29 23:00:55.602093 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:55.602051 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d"] Oct 29 23:00:55.602498 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:00:55.602456 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7a76de_9e42_485e_8034_32d15d55834b.slice/crio-db017ab2419b4035cd1fce58badb267f80f94e62349ef8c6bf5e9b4ea1ac00f2 WatchSource:0}: Error finding container db017ab2419b4035cd1fce58badb267f80f94e62349ef8c6bf5e9b4ea1ac00f2: Status 404 returned error can't find the container with id db017ab2419b4035cd1fce58badb267f80f94e62349ef8c6bf5e9b4ea1ac00f2 Oct 29 23:00:55.714841 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:55.714812 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c" Oct 29 23:00:55.778349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:55.778304 5057 patch_prober.go:29] interesting pod/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:00:55.778511 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:55.778367 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" podUID="9f124749-58f1-491c-b3e8-f1a0d0adb8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:55.817257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:55.817224 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e373476e-bc93-4405-b912-ba19c6636462-kube-api-access-t26l5\" (UniqueName: \"kubernetes.io/projected/e373476e-bc93-4405-b912-ba19c6636462-kube-api-access-t26l5\") pod \"e373476e-bc93-4405-b912-ba19c6636462\" (UID: \"e373476e-bc93-4405-b912-ba19c6636462\") " Oct 29 23:00:55.817413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:55.817368 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e373476e-bc93-4405-b912-ba19c6636462-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e373476e-bc93-4405-b912-ba19c6636462-plugins-config-volume\") pod \"e373476e-bc93-4405-b912-ba19c6636462\" (UID: \"e373476e-bc93-4405-b912-ba19c6636462\") " Oct 29 23:00:55.818179 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:55.818147 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e373476e-bc93-4405-b912-ba19c6636462-plugins-config-volume" pod "e373476e-bc93-4405-b912-ba19c6636462" (UID: "e373476e-bc93-4405-b912-ba19c6636462"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:55.819792 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:55.819761 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e373476e-bc93-4405-b912-ba19c6636462-kube-api-access-t26l5" pod "e373476e-bc93-4405-b912-ba19c6636462" (UID: "e373476e-bc93-4405-b912-ba19c6636462"). InnerVolumeSpecName "kube-api-access-t26l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:55.832813 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:55.832775 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j" podUID="230c87a0-7758-4e6c-bb81-ebd5280d63ad" containerName="payload-pre-processing" containerID="cri-o://cfb3678c9509d8ec2314daa5d550641741fd9bc2fbf5f52fc26a8da9c7a9e056" gracePeriod=25 Oct 29 23:00:55.834142 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:55.833648 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp" podUID="b414763c-3c92-42d4-b2ff-a8514487e68c" containerName="payload-processing" containerID="cri-o://4c561035cfce99a79d78ad87a3e36640e51676c7579eb1e3ef3c6ad0f75592b2" gracePeriod=25 Oct 29 23:00:55.923039 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:55.920338 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e373476e-bc93-4405-b912-ba19c6636462-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:55.923039 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:55.920370 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t26l5\" (UniqueName: \"kubernetes.io/projected/e373476e-bc93-4405-b912-ba19c6636462-kube-api-access-t26l5\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:56.312085 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.312053 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp" Oct 29 23:00:56.328305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.327058 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b414763c-3c92-42d4-b2ff-a8514487e68c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b414763c-3c92-42d4-b2ff-a8514487e68c-plugins-config-volume\") pod \"b414763c-3c92-42d4-b2ff-a8514487e68c\" (UID: \"b414763c-3c92-42d4-b2ff-a8514487e68c\") " Oct 29 23:00:56.328305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.327257 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b414763c-3c92-42d4-b2ff-a8514487e68c-kube-api-access-f7nth\" (UniqueName: \"kubernetes.io/projected/b414763c-3c92-42d4-b2ff-a8514487e68c-kube-api-access-f7nth\") pod \"b414763c-3c92-42d4-b2ff-a8514487e68c\" (UID: \"b414763c-3c92-42d4-b2ff-a8514487e68c\") " Oct 29 23:00:56.328305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.327653 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b414763c-3c92-42d4-b2ff-a8514487e68c-plugins-config-volume" pod "b414763c-3c92-42d4-b2ff-a8514487e68c" (UID: "b414763c-3c92-42d4-b2ff-a8514487e68c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:56.328305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.328052 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b414763c-3c92-42d4-b2ff-a8514487e68c-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:56.332141 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.332103 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b414763c-3c92-42d4-b2ff-a8514487e68c-kube-api-access-f7nth" pod "b414763c-3c92-42d4-b2ff-a8514487e68c" (UID: "b414763c-3c92-42d4-b2ff-a8514487e68c"). InnerVolumeSpecName "kube-api-access-f7nth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:56.341771 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.341677 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm" event={"ID":"7c2efb3f-9de1-42d2-9a4e-ecc08f995e07","Type":"ContainerStarted","Data":"3a58eae9f55542e699b20272280e2f56b5fcc3d9d80d3a6f867373bbc0631554"} Oct 29 23:00:56.348465 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.348379 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d" event={"ID":"ed7a76de-9e42-485e-8034-32d15d55834b","Type":"ContainerStarted","Data":"db017ab2419b4035cd1fce58badb267f80f94e62349ef8c6bf5e9b4ea1ac00f2"} Oct 29 23:00:56.351613 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.351136 5057 generic.go:356] "Generic (PLEG): container finished" podID="e373476e-bc93-4405-b912-ba19c6636462" containerID="8e96911d78fd28730c08e1c3fc31fe2945f761d66970b160ffbff6cd8bca4670" exitCode=0 Oct 29 23:00:56.351613 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.351237 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c" event={"ID":"e373476e-bc93-4405-b912-ba19c6636462","Type":"ContainerDied","Data":"8e96911d78fd28730c08e1c3fc31fe2945f761d66970b160ffbff6cd8bca4670"} Oct 29 23:00:56.351613 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.351279 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c" event={"ID":"e373476e-bc93-4405-b912-ba19c6636462","Type":"ContainerDied","Data":"b4e66f73203899fd8f94f91f4cb9fade2c59673198bd2363cc81b586359362f7"} Oct 29 23:00:56.351613 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.351309 5057 scope.go:122] "RemoveContainer" containerID="8e96911d78fd28730c08e1c3fc31fe2945f761d66970b160ffbff6cd8bca4670" Oct 29 23:00:56.351613 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.351459 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c" Oct 29 23:00:56.388362 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.382831 5057 generic.go:356] "Generic (PLEG): container finished" podID="b414763c-3c92-42d4-b2ff-a8514487e68c" containerID="4c561035cfce99a79d78ad87a3e36640e51676c7579eb1e3ef3c6ad0f75592b2" exitCode=0 Oct 29 23:00:56.388362 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.382946 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp" event={"ID":"b414763c-3c92-42d4-b2ff-a8514487e68c","Type":"ContainerDied","Data":"4c561035cfce99a79d78ad87a3e36640e51676c7579eb1e3ef3c6ad0f75592b2"} Oct 29 23:00:56.388362 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.382996 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp" event={"ID":"b414763c-3c92-42d4-b2ff-a8514487e68c","Type":"ContainerDied","Data":"44926cc2db0777680bf6e1ee366857b83b777b66255ea505b1e0c7919d8dbc8a"} Oct 29 23:00:56.388362 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.383086 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp" Oct 29 23:00:56.393836 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.391268 5057 generic.go:356] "Generic (PLEG): container finished" podID="230c87a0-7758-4e6c-bb81-ebd5280d63ad" containerID="cfb3678c9509d8ec2314daa5d550641741fd9bc2fbf5f52fc26a8da9c7a9e056" exitCode=0 Oct 29 23:00:56.393836 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.391457 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j" event={"ID":"230c87a0-7758-4e6c-bb81-ebd5280d63ad","Type":"ContainerDied","Data":"cfb3678c9509d8ec2314daa5d550641741fd9bc2fbf5f52fc26a8da9c7a9e056"} Oct 29 23:00:56.396130 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.396105 5057 scope.go:122] "RemoveContainer" containerID="8e96911d78fd28730c08e1c3fc31fe2945f761d66970b160ffbff6cd8bca4670" Oct 29 23:00:56.397208 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:00:56.397039 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e96911d78fd28730c08e1c3fc31fe2945f761d66970b160ffbff6cd8bca4670\": container with ID starting with 8e96911d78fd28730c08e1c3fc31fe2945f761d66970b160ffbff6cd8bca4670 not found: ID does not exist" containerID="8e96911d78fd28730c08e1c3fc31fe2945f761d66970b160ffbff6cd8bca4670" Oct 29 23:00:56.397208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.397079 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e96911d78fd28730c08e1c3fc31fe2945f761d66970b160ffbff6cd8bca4670"} err="failed to get container status \"8e96911d78fd28730c08e1c3fc31fe2945f761d66970b160ffbff6cd8bca4670\": rpc error: code = NotFound desc = could not find container \"8e96911d78fd28730c08e1c3fc31fe2945f761d66970b160ffbff6cd8bca4670\": container with ID starting with 8e96911d78fd28730c08e1c3fc31fe2945f761d66970b160ffbff6cd8bca4670 not found: ID does not exist" Oct 29 23:00:56.397208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.397110 5057 scope.go:122] "RemoveContainer" containerID="4c561035cfce99a79d78ad87a3e36640e51676c7579eb1e3ef3c6ad0f75592b2" Oct 29 23:00:56.398241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.398110 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j" Oct 29 23:00:56.458275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.457266 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230c87a0-7758-4e6c-bb81-ebd5280d63ad-kube-api-access-sqzx7\" (UniqueName: \"kubernetes.io/projected/230c87a0-7758-4e6c-bb81-ebd5280d63ad-kube-api-access-sqzx7\") pod \"230c87a0-7758-4e6c-bb81-ebd5280d63ad\" (UID: \"230c87a0-7758-4e6c-bb81-ebd5280d63ad\") " Oct 29 23:00:56.458275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.457311 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230c87a0-7758-4e6c-bb81-ebd5280d63ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230c87a0-7758-4e6c-bb81-ebd5280d63ad-plugins-config-volume\") pod \"230c87a0-7758-4e6c-bb81-ebd5280d63ad\" (UID: \"230c87a0-7758-4e6c-bb81-ebd5280d63ad\") " Oct 29 23:00:56.458275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.457932 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7nth\" (UniqueName: \"kubernetes.io/projected/b414763c-3c92-42d4-b2ff-a8514487e68c-kube-api-access-f7nth\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:56.459654 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.459019 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230c87a0-7758-4e6c-bb81-ebd5280d63ad-plugins-config-volume" pod "230c87a0-7758-4e6c-bb81-ebd5280d63ad" (UID: "230c87a0-7758-4e6c-bb81-ebd5280d63ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:56.460741 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.460724 5057 scope.go:122] "RemoveContainer" containerID="4c561035cfce99a79d78ad87a3e36640e51676c7579eb1e3ef3c6ad0f75592b2" Oct 29 23:00:56.462945 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:00:56.462867 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c561035cfce99a79d78ad87a3e36640e51676c7579eb1e3ef3c6ad0f75592b2\": container with ID starting with 4c561035cfce99a79d78ad87a3e36640e51676c7579eb1e3ef3c6ad0f75592b2 not found: ID does not exist" containerID="4c561035cfce99a79d78ad87a3e36640e51676c7579eb1e3ef3c6ad0f75592b2" Oct 29 23:00:56.462945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.462905 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c561035cfce99a79d78ad87a3e36640e51676c7579eb1e3ef3c6ad0f75592b2"} err="failed to get container status \"4c561035cfce99a79d78ad87a3e36640e51676c7579eb1e3ef3c6ad0f75592b2\": rpc error: code = NotFound desc = could not find container \"4c561035cfce99a79d78ad87a3e36640e51676c7579eb1e3ef3c6ad0f75592b2\": container with ID starting with 4c561035cfce99a79d78ad87a3e36640e51676c7579eb1e3ef3c6ad0f75592b2 not found: ID does not exist" Oct 29 23:00:56.480910 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.480880 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230c87a0-7758-4e6c-bb81-ebd5280d63ad-kube-api-access-sqzx7" pod "230c87a0-7758-4e6c-bb81-ebd5280d63ad" (UID: "230c87a0-7758-4e6c-bb81-ebd5280d63ad"). InnerVolumeSpecName "kube-api-access-sqzx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:56.484728 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.484670 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c"] Oct 29 23:00:56.490382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.490334 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-6dcf86956c-g255c"] Oct 29 23:00:56.560616 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.560294 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqzx7\" (UniqueName: \"kubernetes.io/projected/230c87a0-7758-4e6c-bb81-ebd5280d63ad-kube-api-access-sqzx7\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:56.560616 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.560327 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230c87a0-7758-4e6c-bb81-ebd5280d63ad-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:00:56.610275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.610238 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp"] Oct 29 23:00:56.620283 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:56.620242 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8b57a2-8bf8b5446-5vzmp"] Oct 29 23:00:57.478969 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:57.478937 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j" Oct 29 23:00:57.496439 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:57.496403 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b414763c-3c92-42d4-b2ff-a8514487e68c" path="/var/lib/kubelet/pods/b414763c-3c92-42d4-b2ff-a8514487e68c/volumes" Oct 29 23:00:57.501691 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:57.501664 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e373476e-bc93-4405-b912-ba19c6636462" path="/var/lib/kubelet/pods/e373476e-bc93-4405-b912-ba19c6636462/volumes" Oct 29 23:00:57.502712 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:57.502689 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j" event={"ID":"230c87a0-7758-4e6c-bb81-ebd5280d63ad","Type":"ContainerDied","Data":"75686c1721b92a70d427d1764fcaf78ba42ccdb582cdfbb4e0d590ebe300f686"} Oct 29 23:00:57.502887 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:57.502869 5057 scope.go:122] "RemoveContainer" containerID="cfb3678c9509d8ec2314daa5d550641741fd9bc2fbf5f52fc26a8da9c7a9e056" Oct 29 23:00:57.563310 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:57.563031 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j"] Oct 29 23:00:57.576401 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:57.576369 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8b57a2-58675d7647-66f8j"] Oct 29 23:00:59.486599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:00:59.486556 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230c87a0-7758-4e6c-bb81-ebd5280d63ad" path="/var/lib/kubelet/pods/230c87a0-7758-4e6c-bb81-ebd5280d63ad/volumes" Oct 29 23:01:00.624094 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:00.514698 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d" event={"ID":"ed7a76de-9e42-485e-8034-32d15d55834b","Type":"ContainerStarted","Data":"a2ebc2bd2199d832cffcf60639d66dd4a6386acd10ca13f1963c868da680fbac"} Oct 29 23:01:00.624094 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:00.515142 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d" Oct 29 23:01:00.624094 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:00.517854 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm" event={"ID":"7c2efb3f-9de1-42d2-9a4e-ecc08f995e07","Type":"ContainerStarted","Data":"a476e7560365873ccf24e1713d650dca45d4d024bd34de3fe2383917d3309878"} Oct 29 23:01:00.624094 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:00.517920 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm" Oct 29 23:01:00.624094 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:00.570251 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d" podStartSLOduration=2.132096019 podStartE2EDuration="6.570233812s" totalImagesPullingTime="4.438137793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:00.564014503 +0000 UTC m=+1107.215513034" watchObservedRunningTime="2026-10-29 23:01:00.570233812 +0000 UTC m=+1107.221732342" Oct 29 23:01:00.624094 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:00.610739 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm" podStartSLOduration=2.163276234 podStartE2EDuration="6.610721688s" totalImagesPullingTime="4.447445454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:00.607849297 +0000 UTC m=+1107.259347829" watchObservedRunningTime="2026-10-29 23:01:00.610721688 +0000 UTC m=+1107.262220224" Oct 29 23:01:01.180106 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:01.180006 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": read tcp 10.132.0.2:33580->10.132.0.114:15021: read: connection reset by peer" start-of-body= Oct 29 23:01:01.180106 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:01.180077 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" podUID="860644cd-397b-4af8-b425-a81491c9b66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": read tcp 10.132.0.2:33580->10.132.0.114:15021: read: connection reset by peer" Oct 29 23:01:01.528014 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:01.527930 5057 generic.go:356] "Generic (PLEG): container finished" podID="9f124749-58f1-491c-b3e8-f1a0d0adb8dc" containerID="82182bc5029c288b02f5fbeeb6f83d83d5efc4d8e3568e39950f8c699877af13" exitCode=0 Oct 29 23:01:01.528174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:01.528004 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" event={"ID":"9f124749-58f1-491c-b3e8-f1a0d0adb8dc","Type":"ContainerDied","Data":"82182bc5029c288b02f5fbeeb6f83d83d5efc4d8e3568e39950f8c699877af13"} Oct 29 23:01:01.528633 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:01.528610 5057 scope.go:122] "RemoveContainer" containerID="82182bc5029c288b02f5fbeeb6f83d83d5efc4d8e3568e39950f8c699877af13" Oct 29 23:01:01.530086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:01.530064 5057 generic.go:356] "Generic (PLEG): container finished" podID="860644cd-397b-4af8-b425-a81491c9b66b" containerID="0c141101a46c5d3ae43fd187abb8deabb90fa5e28d66d5dbc51f1096d107cd7f" exitCode=0 Oct 29 23:01:01.530170 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:01.530143 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" event={"ID":"860644cd-397b-4af8-b425-a81491c9b66b","Type":"ContainerDied","Data":"0c141101a46c5d3ae43fd187abb8deabb90fa5e28d66d5dbc51f1096d107cd7f"} Oct 29 23:01:01.530776 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:01.530738 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:01.530872 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:01.530832 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:01.530872 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:01.530763 5057 scope.go:122] "RemoveContainer" containerID="0c141101a46c5d3ae43fd187abb8deabb90fa5e28d66d5dbc51f1096d107cd7f" Oct 29 23:01:01.530955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:01.530876 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:01.532590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:01.532545 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:01.532693 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:01.532647 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:01.532741 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:01.532695 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:02.581208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:02.574348 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" event={"ID":"9f124749-58f1-491c-b3e8-f1a0d0adb8dc","Type":"ContainerStarted","Data":"6fb006486828d7a9aeb9263149c6823f408d63081df34bade53ac22b18535453"} Oct 29 23:01:02.590440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:02.590398 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" event={"ID":"860644cd-397b-4af8-b425-a81491c9b66b","Type":"ContainerStarted","Data":"ab9c0e9c18b3005f83902c673219e92108a47242d93426f7163492bca609c233"} Oct 29 23:01:02.787811 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:02.787778 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 23:01:02.787991 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:02.787874 5057 patch_prober.go:29] interesting pod/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 29 23:01:02.787991 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:02.787905 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" podUID="9f124749-58f1-491c-b3e8-f1a0d0adb8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 29 23:01:02.875270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:02.874862 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts"] Oct 29 23:01:02.894214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:02.889200 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b414763c-3c92-42d4-b2ff-a8514487e68c" containerName="payload-processing" Oct 29 23:01:02.894214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:02.889238 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b414763c-3c92-42d4-b2ff-a8514487e68c" containerName="payload-processing" Oct 29 23:01:02.894214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:02.889269 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e373476e-bc93-4405-b912-ba19c6636462" containerName="payload-pre-processing" Oct 29 23:01:02.894214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:02.889277 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e373476e-bc93-4405-b912-ba19c6636462" containerName="payload-pre-processing" Oct 29 23:01:02.894214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:02.889293 5057 cpu_manager.go:414] "containerMap: removing container" podUID="230c87a0-7758-4e6c-bb81-ebd5280d63ad" containerName="payload-pre-processing" Oct 29 23:01:02.894214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:02.889301 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230c87a0-7758-4e6c-bb81-ebd5280d63ad" containerName="payload-pre-processing" Oct 29 23:01:02.894214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:02.889564 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b414763c-3c92-42d4-b2ff-a8514487e68c" containerName="payload-processing" Oct 29 23:01:02.894214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:02.889583 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e373476e-bc93-4405-b912-ba19c6636462" containerName="payload-pre-processing" Oct 29 23:01:02.894214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:02.889595 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="230c87a0-7758-4e6c-bb81-ebd5280d63ad" containerName="payload-pre-processing" Oct 29 23:01:02.990218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:02.980427 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts"] Oct 29 23:01:02.990218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:02.980579 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.032627 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.032598 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 23:01:03.032943 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.032922 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 29 23:01:03.033068 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.033048 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" podUID="860644cd-397b-4af8-b425-a81491c9b66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 29 23:01:03.088219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.084526 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7c486c-openshift-default-dockercfg-trnfr\"" Oct 29 23:01:03.219163 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.219120 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-envoy\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.519515 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.408430 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7" Oct 29 23:01:03.622215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.529638 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-podinfo\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.622215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.606477 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-token\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.622215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.606572 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-workload-socket\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.622215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.606683 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-credential-socket\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.622215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.606936 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-data\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.622215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.607008 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-workload-certs\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.622215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.607215 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39228ac-4129-460c-9a6c-42efc3a207bb-istiod-ca-cert\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.622215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.607266 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxr4f\" (UniqueName: \"kubernetes.io/projected/b39228ac-4129-460c-9a6c-42efc3a207bb-kube-api-access-qxr4f\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.720216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.712155 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39228ac-4129-460c-9a6c-42efc3a207bb-istiod-ca-cert\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.749212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.745934 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxr4f\" (UniqueName: \"kubernetes.io/projected/b39228ac-4129-460c-9a6c-42efc3a207bb-kube-api-access-qxr4f\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.749212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.746046 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-envoy\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.749212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.746170 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-podinfo\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.749212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.746285 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-token\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.749212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.746370 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-workload-socket\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.749212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.746472 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-credential-socket\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.749212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.746526 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-data\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.749212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.746605 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-workload-certs\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.749772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.749682 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39228ac-4129-460c-9a6c-42efc3a207bb-istiod-ca-cert\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.773281 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.752010 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-workload-socket\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.773281 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.752287 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-credential-socket\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.773281 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.752538 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-data\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.798066 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.798030 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-envoy\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.798263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.798204 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-workload-certs\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.802839 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.802770 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-podinfo\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.819738 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.808480 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-token\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.936207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.924321 5057 patch_prober.go:29] interesting pod/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 29 23:01:03.936207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.924381 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" podUID="9f124749-58f1-491c-b3e8-f1a0d0adb8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 29 23:01:03.936207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.934152 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxr4f\" (UniqueName: \"kubernetes.io/projected/b39228ac-4129-460c-9a6c-42efc3a207bb-kube-api-access-qxr4f\") pod \"e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:03.938307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.938178 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 29 23:01:03.938307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:03.938248 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" podUID="860644cd-397b-4af8-b425-a81491c9b66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 29 23:01:04.100250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:04.099430 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:04.173214 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:01:04.172463 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:01:04.294919 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:04.294889 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q" Oct 29 23:01:04.295072 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:04.294977 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv" Oct 29 23:01:04.607250 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:01:04.600938 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39228ac_4129_460c_9a6c_42efc3a207bb.slice/crio-654a3a762912fab726c2c8c230dafc892ce74c506e3e3ac20cf180d13d186d27 WatchSource:0}: Error finding container 654a3a762912fab726c2c8c230dafc892ce74c506e3e3ac20cf180d13d186d27: Status 404 returned error can't find the container with id 654a3a762912fab726c2c8c230dafc892ce74c506e3e3ac20cf180d13d186d27 Oct 29 23:01:04.609404 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:04.609317 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts"] Oct 29 23:01:04.610521 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:04.610485 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:04.610677 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:04.610565 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:04.610677 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:04.610622 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:04.789247 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:04.789209 5057 patch_prober.go:29] interesting pod/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 29 23:01:04.789811 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:04.789779 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" podUID="9f124749-58f1-491c-b3e8-f1a0d0adb8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 29 23:01:04.890427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:04.890044 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" event={"ID":"b39228ac-4129-460c-9a6c-42efc3a207bb","Type":"ContainerStarted","Data":"b76a1dead114db83f75eb9643df5f0ed2b3bcf297a1746b64661dd1ebfe04460"} Oct 29 23:01:04.890427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:04.890117 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" event={"ID":"b39228ac-4129-460c-9a6c-42efc3a207bb","Type":"ContainerStarted","Data":"654a3a762912fab726c2c8c230dafc892ce74c506e3e3ac20cf180d13d186d27"} Oct 29 23:01:04.952461 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:04.943785 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 29 23:01:04.952461 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:04.943842 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" podUID="860644cd-397b-4af8-b425-a81491c9b66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 29 23:01:04.977212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:04.977106 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" podStartSLOduration=2.977084579 podStartE2EDuration="2.977084579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:04.97368845 +0000 UTC m=+1111.625186995" watchObservedRunningTime="2026-10-29 23:01:04.977084579 +0000 UTC m=+1111.628583110" Oct 29 23:01:05.113309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:05.110848 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:05.113309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:05.110996 5057 patch_prober.go:29] interesting pod/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 23:01:05.113309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:05.111040 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" podUID="b39228ac-4129-460c-9a6c-42efc3a207bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 23:01:05.318780 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:05.318407 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np" Oct 29 23:01:05.787693 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:05.787659 5057 patch_prober.go:29] interesting pod/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 29 23:01:05.788353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:05.788319 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" podUID="9f124749-58f1-491c-b3e8-f1a0d0adb8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 29 23:01:05.939208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:05.937546 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 29 23:01:05.939208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:05.937603 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" podUID="860644cd-397b-4af8-b425-a81491c9b66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 29 23:01:06.110239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:06.110148 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:06.786974 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:06.786928 5057 patch_prober.go:29] interesting pod/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 29 23:01:06.787137 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:06.786984 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" podUID="9f124749-58f1-491c-b3e8-f1a0d0adb8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 29 23:01:06.918148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:06.918112 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:06.920199 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:06.920157 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:01:06.949212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:06.943585 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 29 23:01:06.949212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:06.943642 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" podUID="860644cd-397b-4af8-b425-a81491c9b66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 29 23:01:07.213338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.213300 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf"] Oct 29 23:01:07.219275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.219251 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj"] Oct 29 23:01:07.219557 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.219532 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.280222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.248507 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-660540-openshift-default-dockercfg-qw84q\"" Oct 29 23:01:07.299514 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.299478 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf"] Oct 29 23:01:07.299672 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.299611 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.300280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.300257 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj"] Oct 29 23:01:07.410989 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.410953 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.410989 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.410994 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.411295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.411098 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.411295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.411198 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.411295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.411252 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.411401 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.411375 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.411433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.411404 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a46d30-1423-40eb-b6ec-00345bfd910a-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.411485 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.411472 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.411533 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.411522 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.411608 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.411592 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhvf4\" (UniqueName: \"kubernetes.io/projected/a2f1b565-3ffb-4859-9284-024f97d2f84a-kube-api-access-vhvf4\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.411673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.411633 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.411673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.411660 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.411763 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.411694 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.411763 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.411753 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f1b565-3ffb-4859-9284-024f97d2f84a-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.411828 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.411802 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.411828 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.411820 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.411892 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.411868 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v222t\" (UniqueName: \"kubernetes.io/projected/17a46d30-1423-40eb-b6ec-00345bfd910a-kube-api-access-v222t\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.411921 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.411905 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.513377 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.513278 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.513552 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.513387 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.513552 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.513411 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.513891 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.513864 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.513982 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.513866 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.513982 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.513920 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.513982 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.513961 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.514093 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.514013 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.514150 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.514131 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.514183 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.514165 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a46d30-1423-40eb-b6ec-00345bfd910a-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.514261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.514240 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.514261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.514251 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.514356 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.514336 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.514392 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.514373 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhvf4\" (UniqueName: \"kubernetes.io/projected/a2f1b565-3ffb-4859-9284-024f97d2f84a-kube-api-access-vhvf4\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.514426 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.514405 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.514495 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.514431 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.514495 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.514471 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.514495 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.514485 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.514620 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.514543 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f1b565-3ffb-4859-9284-024f97d2f84a-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.514620 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.514607 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.514714 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.514636 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.514763 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.514731 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v222t\" (UniqueName: \"kubernetes.io/projected/17a46d30-1423-40eb-b6ec-00345bfd910a-kube-api-access-v222t\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.514884 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.514859 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.515011 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.514959 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.515863 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.515557 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.515863 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.515789 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f1b565-3ffb-4859-9284-024f97d2f84a-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.516311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.516260 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.516311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.516287 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.516456 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.516322 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a46d30-1423-40eb-b6ec-00345bfd910a-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.517076 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.517053 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.517337 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.517318 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.518545 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.518510 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.547001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.546684 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.554672 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.552855 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhvf4\" (UniqueName: \"kubernetes.io/projected/a2f1b565-3ffb-4859-9284-024f97d2f84a-kube-api-access-vhvf4\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.554672 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.554622 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.582660 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.581749 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v222t\" (UniqueName: \"kubernetes.io/projected/17a46d30-1423-40eb-b6ec-00345bfd910a-kube-api-access-v222t\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.609358 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.609320 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:07.644648 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.644580 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:07.787704 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.787618 5057 patch_prober.go:29] interesting pod/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 29 23:01:07.787704 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:07.787681 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" podUID="9f124749-58f1-491c-b3e8-f1a0d0adb8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 29 23:01:08.122257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:08.108305 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 29 23:01:08.122257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:08.108387 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" podUID="860644cd-397b-4af8-b425-a81491c9b66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 29 23:01:08.708603 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:01:08.708568 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a46d30_1423_40eb_b6ec_00345bfd910a.slice/crio-7f22317126c86c1a539368bf1a7b6b1275da6d7a41c57393158adaab8e1e63df WatchSource:0}: Error finding container 7f22317126c86c1a539368bf1a7b6b1275da6d7a41c57393158adaab8e1e63df: Status 404 returned error can't find the container with id 7f22317126c86c1a539368bf1a7b6b1275da6d7a41c57393158adaab8e1e63df Oct 29 23:01:08.713879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:08.713834 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:08.715038 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:08.714564 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:08.715739 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:08.715711 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:08.717993 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:08.714474 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf"] Oct 29 23:01:08.717993 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:01:08.714442 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f1b565_3ffb_4859_9284_024f97d2f84a.slice/crio-8db1ee082b733202ea97e529d1c6fac81689e2e6812f7f27ddd09632350da1a2 WatchSource:0}: Error finding container 8db1ee082b733202ea97e529d1c6fac81689e2e6812f7f27ddd09632350da1a2: Status 404 returned error can't find the container with id 8db1ee082b733202ea97e529d1c6fac81689e2e6812f7f27ddd09632350da1a2 Oct 29 23:01:08.720544 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:08.720515 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:08.720722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:08.720701 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:08.720847 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:08.720831 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:08.781568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:08.759123 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj"] Oct 29 23:01:08.793531 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:08.793504 5057 patch_prober.go:29] interesting pod/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 29 23:01:08.793712 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:08.793694 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" podUID="9f124749-58f1-491c-b3e8-f1a0d0adb8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 29 23:01:08.942226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:08.937555 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 29 23:01:08.942226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:08.937613 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" podUID="860644cd-397b-4af8-b425-a81491c9b66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 29 23:01:09.225096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:09.223890 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" event={"ID":"a2f1b565-3ffb-4859-9284-024f97d2f84a","Type":"ContainerStarted","Data":"d371450bc265a3437b4cd04b19a6d8e8feaf6fd9ef4358d61203996e3a2e17f6"} Oct 29 23:01:09.225096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:09.223963 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" event={"ID":"a2f1b565-3ffb-4859-9284-024f97d2f84a","Type":"ContainerStarted","Data":"8db1ee082b733202ea97e529d1c6fac81689e2e6812f7f27ddd09632350da1a2"} Oct 29 23:01:09.259655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:09.259558 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" event={"ID":"17a46d30-1423-40eb-b6ec-00345bfd910a","Type":"ContainerStarted","Data":"75eeba9b629322a33e1a8527186808554ac25ce56d172d229e8df79e27edb3af"} Oct 29 23:01:09.259655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:09.259626 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" event={"ID":"17a46d30-1423-40eb-b6ec-00345bfd910a","Type":"ContainerStarted","Data":"7f22317126c86c1a539368bf1a7b6b1275da6d7a41c57393158adaab8e1e63df"} Oct 29 23:01:09.408124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:09.393511 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" podStartSLOduration=2.393487806 podStartE2EDuration="2.393487806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:09.295692886 +0000 UTC m=+1115.947191422" watchObservedRunningTime="2026-10-29 23:01:09.393487806 +0000 UTC m=+1116.044986336" Oct 29 23:01:09.458731 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:09.457864 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" podStartSLOduration=2.457841057 podStartE2EDuration="2.457841057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:09.368133926 +0000 UTC m=+1116.019632457" watchObservedRunningTime="2026-10-29 23:01:09.457841057 +0000 UTC m=+1116.109339592" Oct 29 23:01:09.664310 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:09.654573 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:09.677238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:09.674358 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:09.690207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:09.680964 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 29 23:01:09.690207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:09.681029 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" podUID="17a46d30-1423-40eb-b6ec-00345bfd910a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 29 23:01:09.708247 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:09.693406 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:01:09.708247 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:09.693465 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" podUID="a2f1b565-3ffb-4859-9284-024f97d2f84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 29 23:01:09.750237 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:09.746944 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 23:01:09.833237 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:09.831272 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 23:01:10.056627 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:10.056591 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 23:01:10.078363 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:10.078336 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 23:01:10.101527 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:10.101491 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 23:01:10.318449 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:10.315422 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 23:01:10.661256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:10.611543 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 29 23:01:10.661256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:10.627271 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" podUID="17a46d30-1423-40eb-b6ec-00345bfd910a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 29 23:01:10.686912 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:10.686876 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:01:10.687736 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:10.687700 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" podUID="a2f1b565-3ffb-4859-9284-024f97d2f84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 29 23:01:11.648213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:11.635919 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 29 23:01:11.648213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:11.635984 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" podUID="17a46d30-1423-40eb-b6ec-00345bfd910a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 29 23:01:11.651034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:11.651000 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:01:11.651311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:11.651266 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" podUID="a2f1b565-3ffb-4859-9284-024f97d2f84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 29 23:01:11.701724 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:11.701692 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d" Oct 29 23:01:11.721688 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:11.721662 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm" Oct 29 23:01:12.609837 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:12.609799 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 29 23:01:12.610038 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:12.609857 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" podUID="17a46d30-1423-40eb-b6ec-00345bfd910a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 29 23:01:12.647261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:12.647225 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:01:12.647512 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:12.647486 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" podUID="a2f1b565-3ffb-4859-9284-024f97d2f84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 29 23:01:13.609843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:13.609803 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 29 23:01:13.610301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:13.609864 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" podUID="17a46d30-1423-40eb-b6ec-00345bfd910a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 29 23:01:13.669995 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:13.669957 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:01:13.670250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:13.670221 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" podUID="a2f1b565-3ffb-4859-9284-024f97d2f84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 29 23:01:14.613316 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:14.611228 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 29 23:01:14.613316 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:14.611304 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" podUID="17a46d30-1423-40eb-b6ec-00345bfd910a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 29 23:01:14.697791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:14.640259 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq"] Oct 29 23:01:14.697791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:14.690552 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:01:14.697791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:14.690616 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" podUID="a2f1b565-3ffb-4859-9284-024f97d2f84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 29 23:01:14.702756 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:14.702726 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq"] Oct 29 23:01:14.702922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:14.702845 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:14.976219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:14.756398 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-50734e-openshift-default-dockercfg-wjnvl\"" Oct 29 23:01:14.976219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:14.886089 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-token\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.397577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.387414 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-envoy\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.397577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.387462 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-workload-socket\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.397577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.387506 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cf5e014-85ba-4024-9ebc-088735098c5c-istiod-ca-cert\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.397577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.387561 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-workload-certs\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.397577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.387793 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-credential-socket\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.397577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.387899 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-podinfo\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.397577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.387967 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-data\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.397577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.388042 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqzmp\" (UniqueName: \"kubernetes.io/projected/7cf5e014-85ba-4024-9ebc-088735098c5c-kube-api-access-sqzmp\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.564301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.552275 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-envoy\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.564301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.552323 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-workload-socket\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.564301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.552378 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cf5e014-85ba-4024-9ebc-088735098c5c-istiod-ca-cert\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.564301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.552431 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-workload-certs\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.564301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.552634 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-credential-socket\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.564301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.552699 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-podinfo\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.564301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.552752 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-data\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.564301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.552813 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqzmp\" (UniqueName: \"kubernetes.io/projected/7cf5e014-85ba-4024-9ebc-088735098c5c-kube-api-access-sqzmp\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.564301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.552961 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-token\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.564301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.556982 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-credential-socket\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.564301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.557266 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-workload-socket\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.564301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.557961 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cf5e014-85ba-4024-9ebc-088735098c5c-istiod-ca-cert\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.564301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.558261 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-workload-certs\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.564301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.558743 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-data\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.617211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.594021 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-envoy\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.641681 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.641637 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 29 23:01:15.641880 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.641696 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" podUID="17a46d30-1423-40eb-b6ec-00345bfd910a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 29 23:01:15.700673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.695753 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:01:15.700673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.695818 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" podUID="a2f1b565-3ffb-4859-9284-024f97d2f84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 29 23:01:15.788268 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.785160 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 23:01:15.788268 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.787804 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-podinfo\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.792790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.792729 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-token\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.818067 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.818033 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqzmp\" (UniqueName: \"kubernetes.io/projected/7cf5e014-85ba-4024-9ebc-088735098c5c-kube-api-access-sqzmp\") pod \"e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:15.883982 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.883939 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:01:15.945443 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:15.945402 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:16.589207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:16.585455 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:16.589207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:16.585561 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:16.589207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:16.585608 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:16.592468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:16.592439 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq"] Oct 29 23:01:16.611966 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:16.610512 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 29 23:01:16.611966 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:16.610565 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" podUID="17a46d30-1423-40eb-b6ec-00345bfd910a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 29 23:01:16.650270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:16.650114 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:01:16.650270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:16.650169 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" podUID="a2f1b565-3ffb-4859-9284-024f97d2f84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 29 23:01:16.697942 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:16.697883 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" event={"ID":"7cf5e014-85ba-4024-9ebc-088735098c5c","Type":"ContainerStarted","Data":"10ebdf4740e472cbca5c1e9406dd54176a12be46aa11cb1bc67dd5585538f0fc"} Oct 29 23:01:17.725617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:17.724690 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" event={"ID":"7cf5e014-85ba-4024-9ebc-088735098c5c","Type":"ContainerStarted","Data":"834443d9a86fb7261754ef952239745c80d8ff100b89cffcbebc26741059c880"} Oct 29 23:01:17.745112 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:17.742717 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:17.745112 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:17.744288 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:17.751450 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:17.751425 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:01:17.759465 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:17.759303 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd"] Oct 29 23:01:18.096281 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.088624 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" podStartSLOduration=4.08860269 podStartE2EDuration="4.08860269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:17.793015588 +0000 UTC m=+1124.444514118" watchObservedRunningTime="2026-10-29 23:01:18.08860269 +0000 UTC m=+1124.740101220" Oct 29 23:01:18.100049 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.100019 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:18.100316 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.100294 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd"] Oct 29 23:01:18.100557 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.100542 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:18.100699 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.100689 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:18.127591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.127328 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:18.127591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.127487 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd" Oct 29 23:01:18.139860 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.139835 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:01:18.149422 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.149396 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-475876\"" Oct 29 23:01:18.149721 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.149703 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-475876-dockercfg-ck8fw\"" Oct 29 23:01:18.260118 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.260083 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7241fa67-618b-4720-a365-9bf989b92131-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd\" (UID: \"7241fa67-618b-4720-a365-9bf989b92131\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd" Oct 29 23:01:18.260699 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.260674 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r2qx\" (UniqueName: \"kubernetes.io/projected/7241fa67-618b-4720-a365-9bf989b92131-kube-api-access-9r2qx\") pod \"payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd\" (UID: \"7241fa67-618b-4720-a365-9bf989b92131\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd" Oct 29 23:01:18.266059 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.266031 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq"] Oct 29 23:01:18.272551 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.272491 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq" Oct 29 23:01:18.273140 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.273120 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq"] Oct 29 23:01:18.377980 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.371975 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9jmh\" (UniqueName: \"kubernetes.io/projected/a06ccf8d-5d83-474f-8528-e900ee22f687-kube-api-access-s9jmh\") pod \"payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq\" (UID: \"a06ccf8d-5d83-474f-8528-e900ee22f687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq" Oct 29 23:01:18.377980 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.372113 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06ccf8d-5d83-474f-8528-e900ee22f687-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq\" (UID: \"a06ccf8d-5d83-474f-8528-e900ee22f687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq" Oct 29 23:01:18.377980 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.372556 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7241fa67-618b-4720-a365-9bf989b92131-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd\" (UID: \"7241fa67-618b-4720-a365-9bf989b92131\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd" Oct 29 23:01:18.377980 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.372631 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r2qx\" (UniqueName: \"kubernetes.io/projected/7241fa67-618b-4720-a365-9bf989b92131-kube-api-access-9r2qx\") pod \"payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd\" (UID: \"7241fa67-618b-4720-a365-9bf989b92131\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd" Oct 29 23:01:18.377980 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.374099 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7241fa67-618b-4720-a365-9bf989b92131-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd\" (UID: \"7241fa67-618b-4720-a365-9bf989b92131\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd" Oct 29 23:01:18.560208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.555860 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06ccf8d-5d83-474f-8528-e900ee22f687-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq\" (UID: \"a06ccf8d-5d83-474f-8528-e900ee22f687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq" Oct 29 23:01:18.560208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.556398 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jmh\" (UniqueName: \"kubernetes.io/projected/a06ccf8d-5d83-474f-8528-e900ee22f687-kube-api-access-s9jmh\") pod \"payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq\" (UID: \"a06ccf8d-5d83-474f-8528-e900ee22f687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq" Oct 29 23:01:18.560208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.558939 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06ccf8d-5d83-474f-8528-e900ee22f687-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq\" (UID: \"a06ccf8d-5d83-474f-8528-e900ee22f687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq" Oct 29 23:01:18.602496 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.599166 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r2qx\" (UniqueName: \"kubernetes.io/projected/7241fa67-618b-4720-a365-9bf989b92131-kube-api-access-9r2qx\") pod \"payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd\" (UID: \"7241fa67-618b-4720-a365-9bf989b92131\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd" Oct 29 23:01:18.603958 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.603904 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9jmh\" (UniqueName: \"kubernetes.io/projected/a06ccf8d-5d83-474f-8528-e900ee22f687-kube-api-access-s9jmh\") pod \"payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq\" (UID: \"a06ccf8d-5d83-474f-8528-e900ee22f687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq" Oct 29 23:01:18.653289 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.653200 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq" Oct 29 23:01:18.739275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.739068 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:18.740049 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.739998 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:01:18.804866 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.802032 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd" Oct 29 23:01:18.967405 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:01:18.967352 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06ccf8d_5d83_474f_8528_e900ee22f687.slice/crio-1cb91cf276256ba608d229ada29e5a8bc352b948a2b6bf6d1e8c288541653f11 WatchSource:0}: Error finding container 1cb91cf276256ba608d229ada29e5a8bc352b948a2b6bf6d1e8c288541653f11: Status 404 returned error can't find the container with id 1cb91cf276256ba608d229ada29e5a8bc352b948a2b6bf6d1e8c288541653f11 Oct 29 23:01:18.971114 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:18.971087 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq"] Oct 29 23:01:19.268114 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:19.268085 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd"] Oct 29 23:01:19.750964 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:19.750927 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq" event={"ID":"a06ccf8d-5d83-474f-8528-e900ee22f687","Type":"ContainerStarted","Data":"e64fe5c37c4bac65f119fc8f8fffa5703b59bc3374711e3dcb20c47b93867ed4"} Oct 29 23:01:19.751497 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:19.750988 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq" event={"ID":"a06ccf8d-5d83-474f-8528-e900ee22f687","Type":"ContainerStarted","Data":"1cb91cf276256ba608d229ada29e5a8bc352b948a2b6bf6d1e8c288541653f11"} Oct 29 23:01:19.751641 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:19.751616 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq" Oct 29 23:01:19.753464 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:19.753435 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd" event={"ID":"7241fa67-618b-4720-a365-9bf989b92131","Type":"ContainerStarted","Data":"232aa9d95f0177adf59df279d4e3c46a9949e3435ec837fc31eb669b1cecfb98"} Oct 29 23:01:19.753577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:19.753493 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd" event={"ID":"7241fa67-618b-4720-a365-9bf989b92131","Type":"ContainerStarted","Data":"bb8260049284ddda0e31b646ab9bd35275ad30fbcef0536e0eed4d0ed13517dc"} Oct 29 23:01:19.798044 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:19.797997 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq" podStartSLOduration=2.79798478 podStartE2EDuration="2.79798478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:19.794577614 +0000 UTC m=+1126.446076144" watchObservedRunningTime="2026-10-29 23:01:19.79798478 +0000 UTC m=+1126.449483310" Oct 29 23:01:19.908342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:19.908266 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd" podStartSLOduration=2.908245241 podStartE2EDuration="2.908245241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:19.885785458 +0000 UTC m=+1126.537283989" watchObservedRunningTime="2026-10-29 23:01:19.908245241 +0000 UTC m=+1126.559743772" Oct 29 23:01:20.766115 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:20.766086 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd" Oct 29 23:01:29.996489 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:29.996452 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz"] Oct 29 23:01:29.997030 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:29.996730 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" podUID="abbf2a7f-653e-4219-8594-bc67fb660a73" containerName="maas-api" containerID="cri-o://a1232f2980ff965b2b67aef0fab1e485fe5892c1364ec553df3522eab1d9030c" gracePeriod=30 Oct 29 23:01:30.772030 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:30.762660 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" Oct 29 23:01:30.775481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:30.775451 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq" Oct 29 23:01:30.964654 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:30.964561 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abbf2a7f-653e-4219-8594-bc67fb660a73-metrics-certs\" (UniqueName: \"kubernetes.io/secret/abbf2a7f-653e-4219-8594-bc67fb660a73-metrics-certs\") pod \"abbf2a7f-653e-4219-8594-bc67fb660a73\" (UID: \"abbf2a7f-653e-4219-8594-bc67fb660a73\") " Oct 29 23:01:30.964956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:30.964935 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abbf2a7f-653e-4219-8594-bc67fb660a73-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abbf2a7f-653e-4219-8594-bc67fb660a73-maas-api-tls\") pod \"abbf2a7f-653e-4219-8594-bc67fb660a73\" (UID: \"abbf2a7f-653e-4219-8594-bc67fb660a73\") " Oct 29 23:01:30.965617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:30.965591 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abbf2a7f-653e-4219-8594-bc67fb660a73-kube-api-access-nkqvz\" (UniqueName: \"kubernetes.io/projected/abbf2a7f-653e-4219-8594-bc67fb660a73-kube-api-access-nkqvz\") pod \"abbf2a7f-653e-4219-8594-bc67fb660a73\" (UID: \"abbf2a7f-653e-4219-8594-bc67fb660a73\") " Oct 29 23:01:30.966282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:30.966260 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abbf2a7f-653e-4219-8594-bc67fb660a73-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbf2a7f-653e-4219-8594-bc67fb660a73-openshift-service-ca\") pod \"abbf2a7f-653e-4219-8594-bc67fb660a73\" (UID: \"abbf2a7f-653e-4219-8594-bc67fb660a73\") " Oct 29 23:01:30.970148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:30.968694 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abbf2a7f-653e-4219-8594-bc67fb660a73-openshift-service-ca" pod "abbf2a7f-653e-4219-8594-bc67fb660a73" (UID: "abbf2a7f-653e-4219-8594-bc67fb660a73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:31.121322 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:31.110298 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbf2a7f-653e-4219-8594-bc67fb660a73-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:01:31.249031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:31.248947 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abbf2a7f-653e-4219-8594-bc67fb660a73-maas-api-tls" pod "abbf2a7f-653e-4219-8594-bc67fb660a73" (UID: "abbf2a7f-653e-4219-8594-bc67fb660a73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:01:31.250404 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:31.250359 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abbf2a7f-653e-4219-8594-bc67fb660a73-metrics-certs" pod "abbf2a7f-653e-4219-8594-bc67fb660a73" (UID: "abbf2a7f-653e-4219-8594-bc67fb660a73"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:01:31.256005 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:31.255970 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbf2a7f-653e-4219-8594-bc67fb660a73-kube-api-access-nkqvz" pod "abbf2a7f-653e-4219-8594-bc67fb660a73" (UID: "abbf2a7f-653e-4219-8594-bc67fb660a73"). InnerVolumeSpecName "kube-api-access-nkqvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:31.285522 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:31.285478 5057 generic.go:356] "Generic (PLEG): container finished" podID="abbf2a7f-653e-4219-8594-bc67fb660a73" containerID="a1232f2980ff965b2b67aef0fab1e485fe5892c1364ec553df3522eab1d9030c" exitCode=0 Oct 29 23:01:31.285801 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:31.285780 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" event={"ID":"abbf2a7f-653e-4219-8594-bc67fb660a73","Type":"ContainerDied","Data":"a1232f2980ff965b2b67aef0fab1e485fe5892c1364ec553df3522eab1d9030c"} Oct 29 23:01:31.285899 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:31.285884 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" event={"ID":"abbf2a7f-653e-4219-8594-bc67fb660a73","Type":"ContainerDied","Data":"e863319854c61d08029bb71614bf123b205f15bbe6ea8a61f7ec936aa7934e83"} Oct 29 23:01:31.285956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:31.285938 5057 scope.go:122] "RemoveContainer" containerID="a1232f2980ff965b2b67aef0fab1e485fe5892c1364ec553df3522eab1d9030c" Oct 29 23:01:31.286205 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:31.286170 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz" Oct 29 23:01:31.300057 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:31.300037 5057 scope.go:122] "RemoveContainer" containerID="a1232f2980ff965b2b67aef0fab1e485fe5892c1364ec553df3522eab1d9030c" Oct 29 23:01:31.300431 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:01:31.300379 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1232f2980ff965b2b67aef0fab1e485fe5892c1364ec553df3522eab1d9030c\": container with ID starting with a1232f2980ff965b2b67aef0fab1e485fe5892c1364ec553df3522eab1d9030c not found: ID does not exist" containerID="a1232f2980ff965b2b67aef0fab1e485fe5892c1364ec553df3522eab1d9030c" Oct 29 23:01:31.300541 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:31.300441 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1232f2980ff965b2b67aef0fab1e485fe5892c1364ec553df3522eab1d9030c"} err="failed to get container status \"a1232f2980ff965b2b67aef0fab1e485fe5892c1364ec553df3522eab1d9030c\": rpc error: code = NotFound desc = could not find container \"a1232f2980ff965b2b67aef0fab1e485fe5892c1364ec553df3522eab1d9030c\": container with ID starting with a1232f2980ff965b2b67aef0fab1e485fe5892c1364ec553df3522eab1d9030c not found: ID does not exist" Oct 29 23:01:31.321370 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:31.321341 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abbf2a7f-653e-4219-8594-bc67fb660a73-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:01:31.321370 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:31.321372 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abbf2a7f-653e-4219-8594-bc67fb660a73-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:01:31.321534 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:31.321386 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkqvz\" (UniqueName: \"kubernetes.io/projected/abbf2a7f-653e-4219-8594-bc67fb660a73-kube-api-access-nkqvz\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:01:31.348753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:31.348708 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz"] Oct 29 23:01:31.353023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:31.352994 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8b57a2-68f4799869-w6tkz"] Oct 29 23:01:31.484854 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:31.484819 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abbf2a7f-653e-4219-8594-bc67fb660a73" path="/var/lib/kubelet/pods/abbf2a7f-653e-4219-8594-bc67fb660a73/volumes" Oct 29 23:01:31.779117 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:31.779085 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd" Oct 29 23:01:33.821517 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:33.821479 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:01:33.983127 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:01:33.983085 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:01:36.218602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.218559 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n"] Oct 29 23:01:36.219713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.219671 5057 cpu_manager.go:414] "containerMap: removing container" podUID="abbf2a7f-653e-4219-8594-bc67fb660a73" containerName="maas-api" Oct 29 23:01:36.219713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.219705 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbf2a7f-653e-4219-8594-bc67fb660a73" containerName="maas-api" Oct 29 23:01:36.219964 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.219949 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbf2a7f-653e-4219-8594-bc67fb660a73" containerName="maas-api" Oct 29 23:01:36.239596 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.239568 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.254523 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.254496 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n"] Oct 29 23:01:36.263772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.263743 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-10b990-openshift-default-dockercfg-nnpxx\"" Oct 29 23:01:36.454985 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.454945 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.472433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.472344 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e45f5c-eae2-4876-ab74-f5949bea38e1-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.510822 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.510786 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.511074 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.511058 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.516458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.516427 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.516683 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.516664 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xb7n\" (UniqueName: \"kubernetes.io/projected/10e45f5c-eae2-4876-ab74-f5949bea38e1-kube-api-access-5xb7n\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.516871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.516855 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.526370 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.526339 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.555818 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.555784 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.678236 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.672677 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.694834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.687323 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.694834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.687465 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e45f5c-eae2-4876-ab74-f5949bea38e1-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.694834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.687598 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.694834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.687670 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.694834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.687864 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.694834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.687902 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xb7n\" (UniqueName: \"kubernetes.io/projected/10e45f5c-eae2-4876-ab74-f5949bea38e1-kube-api-access-5xb7n\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.694834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.687947 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.694834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.688007 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.699374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.678967 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.699374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.698893 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.701073 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.699833 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e45f5c-eae2-4876-ab74-f5949bea38e1-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.701073 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.699991 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.701073 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.701004 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.747206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.747102 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.779577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.762770 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.808453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.808376 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:36.885061 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:36.885027 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xb7n\" (UniqueName: \"kubernetes.io/projected/10e45f5c-eae2-4876-ab74-f5949bea38e1-kube-api-access-5xb7n\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:37.181207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:37.176610 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:37.817674 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:01:37.817632 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e45f5c_eae2_4876_ab74_f5949bea38e1.slice/crio-b5830b5cf2ddaf6a5a98ba380df36c772283b0e46833674813685f1ae32cc8c5 WatchSource:0}: Error finding container b5830b5cf2ddaf6a5a98ba380df36c772283b0e46833674813685f1ae32cc8c5: Status 404 returned error can't find the container with id b5830b5cf2ddaf6a5a98ba380df36c772283b0e46833674813685f1ae32cc8c5 Oct 29 23:01:37.819053 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:37.819027 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n"] Oct 29 23:01:37.820849 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:37.820809 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:37.820951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:37.820877 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:37.820951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:37.820907 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:38.750655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:38.750616 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" event={"ID":"10e45f5c-eae2-4876-ab74-f5949bea38e1","Type":"ContainerStarted","Data":"4f7b40747316cbceabd34ca9c5d18e21a65a2476fc2bac9f61cf0f662e15603f"} Oct 29 23:01:38.750855 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:38.750676 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" event={"ID":"10e45f5c-eae2-4876-ab74-f5949bea38e1","Type":"ContainerStarted","Data":"b5830b5cf2ddaf6a5a98ba380df36c772283b0e46833674813685f1ae32cc8c5"} Oct 29 23:01:38.824806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:38.824751 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" podStartSLOduration=2.8247351099999998 podStartE2EDuration="2.82473511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:38.819902299 +0000 UTC m=+1145.471400828" watchObservedRunningTime="2026-10-29 23:01:38.82473511 +0000 UTC m=+1145.476233640" Oct 29 23:01:39.186079 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:39.181370 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:39.418610 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:39.418567 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:39.761282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:39.761237 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:39.762494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:39.762470 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:01:41.106220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:41.105898 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk"] Oct 29 23:01:41.126389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:41.126353 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb"] Oct 29 23:01:46.106631 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:46.106592 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk" podUID="ee6dd65e-769c-419f-b16c-1b5abf1e879d" containerName="payload-processing" containerID="cri-o://246bd1582a608d369cc7a6d923d44c464705ddd77d6c30eb6ffffb735a4d1a51" gracePeriod=25 Oct 29 23:01:46.128467 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:46.128418 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb" podUID="879eb585-eb10-4ddf-b447-202688d1e3e8" containerName="payload-pre-processing" containerID="cri-o://734c2217620f2d9e5a7df448e80f947724d887a61d657b71241ac3dea16c5ab0" gracePeriod=25 Oct 29 23:01:46.522319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:46.522283 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk" Oct 29 23:01:46.597614 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:46.597584 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee6dd65e-769c-419f-b16c-1b5abf1e879d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6dd65e-769c-419f-b16c-1b5abf1e879d-plugins-config-volume\") pod \"ee6dd65e-769c-419f-b16c-1b5abf1e879d\" (UID: \"ee6dd65e-769c-419f-b16c-1b5abf1e879d\") " Oct 29 23:01:46.597920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:46.597884 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee6dd65e-769c-419f-b16c-1b5abf1e879d-kube-api-access-4hd86\" (UniqueName: \"kubernetes.io/projected/ee6dd65e-769c-419f-b16c-1b5abf1e879d-kube-api-access-4hd86\") pod \"ee6dd65e-769c-419f-b16c-1b5abf1e879d\" (UID: \"ee6dd65e-769c-419f-b16c-1b5abf1e879d\") " Oct 29 23:01:46.599199 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:46.599162 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee6dd65e-769c-419f-b16c-1b5abf1e879d-plugins-config-volume" pod "ee6dd65e-769c-419f-b16c-1b5abf1e879d" (UID: "ee6dd65e-769c-419f-b16c-1b5abf1e879d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:46.601625 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:46.601601 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6dd65e-769c-419f-b16c-1b5abf1e879d-kube-api-access-4hd86" pod "ee6dd65e-769c-419f-b16c-1b5abf1e879d" (UID: "ee6dd65e-769c-419f-b16c-1b5abf1e879d"). InnerVolumeSpecName "kube-api-access-4hd86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:46.624694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:46.624663 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb" Oct 29 23:01:46.699598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:46.699566 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/879eb585-eb10-4ddf-b447-202688d1e3e8-kube-api-access-ldh9p\" (UniqueName: \"kubernetes.io/projected/879eb585-eb10-4ddf-b447-202688d1e3e8-kube-api-access-ldh9p\") pod \"879eb585-eb10-4ddf-b447-202688d1e3e8\" (UID: \"879eb585-eb10-4ddf-b447-202688d1e3e8\") " Oct 29 23:01:46.699789 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:46.699710 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/879eb585-eb10-4ddf-b447-202688d1e3e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879eb585-eb10-4ddf-b447-202688d1e3e8-plugins-config-volume\") pod \"879eb585-eb10-4ddf-b447-202688d1e3e8\" (UID: \"879eb585-eb10-4ddf-b447-202688d1e3e8\") " Oct 29 23:01:46.700228 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:46.700158 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/879eb585-eb10-4ddf-b447-202688d1e3e8-plugins-config-volume" pod "879eb585-eb10-4ddf-b447-202688d1e3e8" (UID: "879eb585-eb10-4ddf-b447-202688d1e3e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:46.700526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:46.700506 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6dd65e-769c-419f-b16c-1b5abf1e879d-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:01:46.700624 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:46.700529 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879eb585-eb10-4ddf-b447-202688d1e3e8-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:01:46.700624 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:46.700542 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hd86\" (UniqueName: \"kubernetes.io/projected/ee6dd65e-769c-419f-b16c-1b5abf1e879d-kube-api-access-4hd86\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:01:46.702483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:46.702445 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/879eb585-eb10-4ddf-b447-202688d1e3e8-kube-api-access-ldh9p" pod "879eb585-eb10-4ddf-b447-202688d1e3e8" (UID: "879eb585-eb10-4ddf-b447-202688d1e3e8"). InnerVolumeSpecName "kube-api-access-ldh9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:46.802645 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:46.802615 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldh9p\" (UniqueName: \"kubernetes.io/projected/879eb585-eb10-4ddf-b447-202688d1e3e8-kube-api-access-ldh9p\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:01:47.071997 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.071920 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr"] Oct 29 23:01:47.072793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.072777 5057 cpu_manager.go:414] "containerMap: removing container" podUID="879eb585-eb10-4ddf-b447-202688d1e3e8" containerName="payload-pre-processing" Oct 29 23:01:47.072852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.072794 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="879eb585-eb10-4ddf-b447-202688d1e3e8" containerName="payload-pre-processing" Oct 29 23:01:47.072852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.072816 5057 cpu_manager.go:414] "containerMap: removing container" podUID="ee6dd65e-769c-419f-b16c-1b5abf1e879d" containerName="payload-processing" Oct 29 23:01:47.072852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.072822 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee6dd65e-769c-419f-b16c-1b5abf1e879d" containerName="payload-processing" Oct 29 23:01:47.072963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.072954 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="879eb585-eb10-4ddf-b447-202688d1e3e8" containerName="payload-pre-processing" Oct 29 23:01:47.072993 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.072969 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee6dd65e-769c-419f-b16c-1b5abf1e879d" containerName="payload-processing" Oct 29 23:01:47.077253 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.077234 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr" Oct 29 23:01:47.094884 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.094863 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-44a4e4\"" Oct 29 23:01:47.094985 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.094892 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-44a4e4-dockercfg-cjr5x\"" Oct 29 23:01:47.116044 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.116011 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr"] Oct 29 23:01:47.152313 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.152273 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd"] Oct 29 23:01:47.162715 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.162690 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd" Oct 29 23:01:47.208423 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.189342 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd"] Oct 29 23:01:47.342116 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.342008 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8ltw\" (UniqueName: \"kubernetes.io/projected/af4639d7-85ad-45ba-8dda-e0b14e4bb396-kube-api-access-r8ltw\") pod \"payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd\" (UID: \"af4639d7-85ad-45ba-8dda-e0b14e4bb396\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd" Oct 29 23:01:47.342487 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.342455 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f0ff64c-6add-4f9c-bb35-2636838d7a10-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr\" (UID: \"0f0ff64c-6add-4f9c-bb35-2636838d7a10\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr" Oct 29 23:01:47.342697 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.342669 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4639d7-85ad-45ba-8dda-e0b14e4bb396-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd\" (UID: \"af4639d7-85ad-45ba-8dda-e0b14e4bb396\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd" Oct 29 23:01:47.343321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.343282 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5thv8\" (UniqueName: \"kubernetes.io/projected/0f0ff64c-6add-4f9c-bb35-2636838d7a10-kube-api-access-5thv8\") pod \"payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr\" (UID: \"0f0ff64c-6add-4f9c-bb35-2636838d7a10\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr" Oct 29 23:01:47.378904 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.378858 5057 generic.go:356] "Generic (PLEG): container finished" podID="879eb585-eb10-4ddf-b447-202688d1e3e8" containerID="734c2217620f2d9e5a7df448e80f947724d887a61d657b71241ac3dea16c5ab0" exitCode=0 Oct 29 23:01:47.379262 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.379073 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb" Oct 29 23:01:47.382244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.382217 5057 generic.go:356] "Generic (PLEG): container finished" podID="ee6dd65e-769c-419f-b16c-1b5abf1e879d" containerID="246bd1582a608d369cc7a6d923d44c464705ddd77d6c30eb6ffffb735a4d1a51" exitCode=0 Oct 29 23:01:47.382698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.382666 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk" Oct 29 23:01:47.406635 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.406591 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb" event={"ID":"879eb585-eb10-4ddf-b447-202688d1e3e8","Type":"ContainerDied","Data":"734c2217620f2d9e5a7df448e80f947724d887a61d657b71241ac3dea16c5ab0"} Oct 29 23:01:47.428384 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.428333 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb" event={"ID":"879eb585-eb10-4ddf-b447-202688d1e3e8","Type":"ContainerDied","Data":"ccd74884992823cfc88d4c41704c02eb12b263e17d4ed5530a86169a85a06831"} Oct 29 23:01:47.428664 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.428633 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk" event={"ID":"ee6dd65e-769c-419f-b16c-1b5abf1e879d","Type":"ContainerDied","Data":"246bd1582a608d369cc7a6d923d44c464705ddd77d6c30eb6ffffb735a4d1a51"} Oct 29 23:01:47.428805 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.428787 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk" event={"ID":"ee6dd65e-769c-419f-b16c-1b5abf1e879d","Type":"ContainerDied","Data":"528cb7dd0760d8ac7c243fb109eb924bc770d0e09088f50dfbaecc24279b1508"} Oct 29 23:01:47.428956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.428942 5057 scope.go:122] "RemoveContainer" containerID="734c2217620f2d9e5a7df448e80f947724d887a61d657b71241ac3dea16c5ab0" Oct 29 23:01:47.533833 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.533801 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5thv8\" (UniqueName: \"kubernetes.io/projected/0f0ff64c-6add-4f9c-bb35-2636838d7a10-kube-api-access-5thv8\") pod \"payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr\" (UID: \"0f0ff64c-6add-4f9c-bb35-2636838d7a10\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr" Oct 29 23:01:47.534340 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.534316 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8ltw\" (UniqueName: \"kubernetes.io/projected/af4639d7-85ad-45ba-8dda-e0b14e4bb396-kube-api-access-r8ltw\") pod \"payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd\" (UID: \"af4639d7-85ad-45ba-8dda-e0b14e4bb396\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd" Oct 29 23:01:47.534535 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.534521 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f0ff64c-6add-4f9c-bb35-2636838d7a10-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr\" (UID: \"0f0ff64c-6add-4f9c-bb35-2636838d7a10\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr" Oct 29 23:01:47.534677 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.534662 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4639d7-85ad-45ba-8dda-e0b14e4bb396-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd\" (UID: \"af4639d7-85ad-45ba-8dda-e0b14e4bb396\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd" Oct 29 23:01:47.542008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.541979 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f0ff64c-6add-4f9c-bb35-2636838d7a10-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr\" (UID: \"0f0ff64c-6add-4f9c-bb35-2636838d7a10\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr" Oct 29 23:01:47.549394 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.548230 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4639d7-85ad-45ba-8dda-e0b14e4bb396-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd\" (UID: \"af4639d7-85ad-45ba-8dda-e0b14e4bb396\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd" Oct 29 23:01:47.770711 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.770639 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8ltw\" (UniqueName: \"kubernetes.io/projected/af4639d7-85ad-45ba-8dda-e0b14e4bb396-kube-api-access-r8ltw\") pod \"payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd\" (UID: \"af4639d7-85ad-45ba-8dda-e0b14e4bb396\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd" Oct 29 23:01:47.798504 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.795908 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5thv8\" (UniqueName: \"kubernetes.io/projected/0f0ff64c-6add-4f9c-bb35-2636838d7a10-kube-api-access-5thv8\") pod \"payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr\" (UID: \"0f0ff64c-6add-4f9c-bb35-2636838d7a10\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr" Oct 29 23:01:47.798504 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.797025 5057 scope.go:122] "RemoveContainer" containerID="734c2217620f2d9e5a7df448e80f947724d887a61d657b71241ac3dea16c5ab0" Oct 29 23:01:47.808376 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:01:47.802422 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"734c2217620f2d9e5a7df448e80f947724d887a61d657b71241ac3dea16c5ab0\": container with ID starting with 734c2217620f2d9e5a7df448e80f947724d887a61d657b71241ac3dea16c5ab0 not found: ID does not exist" containerID="734c2217620f2d9e5a7df448e80f947724d887a61d657b71241ac3dea16c5ab0" Oct 29 23:01:47.808376 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.802463 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"734c2217620f2d9e5a7df448e80f947724d887a61d657b71241ac3dea16c5ab0"} err="failed to get container status \"734c2217620f2d9e5a7df448e80f947724d887a61d657b71241ac3dea16c5ab0\": rpc error: code = NotFound desc = could not find container \"734c2217620f2d9e5a7df448e80f947724d887a61d657b71241ac3dea16c5ab0\": container with ID starting with 734c2217620f2d9e5a7df448e80f947724d887a61d657b71241ac3dea16c5ab0 not found: ID does not exist" Oct 29 23:01:47.808376 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.802489 5057 scope.go:122] "RemoveContainer" containerID="246bd1582a608d369cc7a6d923d44c464705ddd77d6c30eb6ffffb735a4d1a51" Oct 29 23:01:47.808376 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.807496 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb"] Oct 29 23:01:47.808376 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.807528 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-479c2a-9d469f4c5-89wbb"] Oct 29 23:01:47.832816 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.832779 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk"] Oct 29 23:01:47.851970 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.851933 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-479c2a-86c498f4db-p58nk"] Oct 29 23:01:47.853288 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.853259 5057 scope.go:122] "RemoveContainer" containerID="246bd1582a608d369cc7a6d923d44c464705ddd77d6c30eb6ffffb735a4d1a51" Oct 29 23:01:47.861627 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:01:47.861596 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246bd1582a608d369cc7a6d923d44c464705ddd77d6c30eb6ffffb735a4d1a51\": container with ID starting with 246bd1582a608d369cc7a6d923d44c464705ddd77d6c30eb6ffffb735a4d1a51 not found: ID does not exist" containerID="246bd1582a608d369cc7a6d923d44c464705ddd77d6c30eb6ffffb735a4d1a51" Oct 29 23:01:47.861734 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.861638 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246bd1582a608d369cc7a6d923d44c464705ddd77d6c30eb6ffffb735a4d1a51"} err="failed to get container status \"246bd1582a608d369cc7a6d923d44c464705ddd77d6c30eb6ffffb735a4d1a51\": rpc error: code = NotFound desc = could not find container \"246bd1582a608d369cc7a6d923d44c464705ddd77d6c30eb6ffffb735a4d1a51\": container with ID starting with 246bd1582a608d369cc7a6d923d44c464705ddd77d6c30eb6ffffb735a4d1a51 not found: ID does not exist" Oct 29 23:01:47.979678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.979642 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd" Oct 29 23:01:47.992935 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:47.992908 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr" Oct 29 23:01:48.303693 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:01:48.303646 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4639d7_85ad_45ba_8dda_e0b14e4bb396.slice/crio-41e9ef6478e80be1d00b22e90655ae77303da1c0d46e609da40d41fe416d91d9 WatchSource:0}: Error finding container 41e9ef6478e80be1d00b22e90655ae77303da1c0d46e609da40d41fe416d91d9: Status 404 returned error can't find the container with id 41e9ef6478e80be1d00b22e90655ae77303da1c0d46e609da40d41fe416d91d9 Oct 29 23:01:48.304057 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:48.303966 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd"] Oct 29 23:01:48.335413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:48.335388 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr"] Oct 29 23:01:48.336906 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:01:48.336874 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0ff64c_6add_4f9c_bb35_2636838d7a10.slice/crio-39b11d4f8dea9f4ba183b40d9b30b532745574d6c9cd87096f9ee6f0d2cd922b WatchSource:0}: Error finding container 39b11d4f8dea9f4ba183b40d9b30b532745574d6c9cd87096f9ee6f0d2cd922b: Status 404 returned error can't find the container with id 39b11d4f8dea9f4ba183b40d9b30b532745574d6c9cd87096f9ee6f0d2cd922b Oct 29 23:01:48.393286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:48.393240 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd" event={"ID":"af4639d7-85ad-45ba-8dda-e0b14e4bb396","Type":"ContainerStarted","Data":"41e9ef6478e80be1d00b22e90655ae77303da1c0d46e609da40d41fe416d91d9"} Oct 29 23:01:48.395986 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:48.395957 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr" event={"ID":"0f0ff64c-6add-4f9c-bb35-2636838d7a10","Type":"ContainerStarted","Data":"39b11d4f8dea9f4ba183b40d9b30b532745574d6c9cd87096f9ee6f0d2cd922b"} Oct 29 23:01:49.409277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:49.409238 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd" event={"ID":"af4639d7-85ad-45ba-8dda-e0b14e4bb396","Type":"ContainerStarted","Data":"7f5b382f7831beb2e903ab457c4c0946bc7a25e3afc51c492b04b232d00247e4"} Oct 29 23:01:49.409775 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:49.409310 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd" Oct 29 23:01:49.411115 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:49.411087 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr" event={"ID":"0f0ff64c-6add-4f9c-bb35-2636838d7a10","Type":"ContainerStarted","Data":"54d79c3cc3234f521b85adb3f03a11244d1eab43c6e6d4662b0cd68e34f607a0"} Oct 29 23:01:49.411273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:49.411149 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr" Oct 29 23:01:49.455104 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:49.455045 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd" podStartSLOduration=2.455026324 podStartE2EDuration="2.455026324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:49.453297888 +0000 UTC m=+1156.104796420" watchObservedRunningTime="2026-10-29 23:01:49.455026324 +0000 UTC m=+1156.106524856" Oct 29 23:01:49.485029 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:49.484997 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="879eb585-eb10-4ddf-b447-202688d1e3e8" path="/var/lib/kubelet/pods/879eb585-eb10-4ddf-b447-202688d1e3e8/volumes" Oct 29 23:01:49.485567 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:49.485552 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee6dd65e-769c-419f-b16c-1b5abf1e879d" path="/var/lib/kubelet/pods/ee6dd65e-769c-419f-b16c-1b5abf1e879d/volumes" Oct 29 23:01:49.501425 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:49.501380 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr" podStartSLOduration=2.501367545 podStartE2EDuration="2.501367545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:49.496215243 +0000 UTC m=+1156.147713769" watchObservedRunningTime="2026-10-29 23:01:49.501367545 +0000 UTC m=+1156.152866075" Oct 29 23:01:52.252923 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.252885 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk"] Oct 29 23:01:52.258051 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.258029 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.295650 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.295614 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk"] Oct 29 23:01:52.340570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.340530 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.340781 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.340599 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.340781 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.340710 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.340781 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.340745 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.340965 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.340809 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.340965 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.340911 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.340965 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.340929 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.340965 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.340944 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.341172 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.340998 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq9p2\" (UniqueName: \"kubernetes.io/projected/4e7c1f33-f809-4677-b63d-1d8ad1a36551-kube-api-access-mq9p2\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.442833 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.442795 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.443042 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.442905 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.443042 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.443001 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.443042 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.443035 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.443249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.443131 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.443351 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.443331 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.443351 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.443336 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.443481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.443367 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.443481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.443394 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.443481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.443464 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq9p2\" (UniqueName: \"kubernetes.io/projected/4e7c1f33-f809-4677-b63d-1d8ad1a36551-kube-api-access-mq9p2\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.444026 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.443993 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.444337 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.444314 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.444337 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.444331 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.444595 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.444572 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.446133 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.446112 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.447206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.447170 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.469420 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.469385 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.469553 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.469443 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq9p2\" (UniqueName: \"kubernetes.io/projected/4e7c1f33-f809-4677-b63d-1d8ad1a36551-kube-api-access-mq9p2\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.578726 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.578641 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:52.983425 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:01:52.983395 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7c1f33_f809_4677_b63d_1d8ad1a36551.slice/crio-7b27f3a77a780c7f1b7f8788652fc8c812238d066e4c38dca24664750e087a89 WatchSource:0}: Error finding container 7b27f3a77a780c7f1b7f8788652fc8c812238d066e4c38dca24664750e087a89: Status 404 returned error can't find the container with id 7b27f3a77a780c7f1b7f8788652fc8c812238d066e4c38dca24664750e087a89 Oct 29 23:01:52.986284 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.986259 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk"] Oct 29 23:01:52.986871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.986830 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:52.986933 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.986917 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:52.986974 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:52.986963 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:01:53.505864 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:53.505819 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" event={"ID":"4e7c1f33-f809-4677-b63d-1d8ad1a36551","Type":"ContainerStarted","Data":"aaac47fc6c1abd57a240f46aa1cd3446a3c7a6c221694f04f9f0352793622ca9"} Oct 29 23:01:53.506278 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:53.505889 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" event={"ID":"4e7c1f33-f809-4677-b63d-1d8ad1a36551","Type":"ContainerStarted","Data":"7b27f3a77a780c7f1b7f8788652fc8c812238d066e4c38dca24664750e087a89"} Oct 29 23:01:53.571055 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:53.570985 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" podStartSLOduration=1.570966485 podStartE2EDuration="1.570966485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:53.565076971 +0000 UTC m=+1160.216575503" watchObservedRunningTime="2026-10-29 23:01:53.570966485 +0000 UTC m=+1160.222465016" Oct 29 23:01:53.578890 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:53.578850 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:01:54.580877 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:54.580841 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 29 23:01:54.581424 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:54.580905 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" podUID="4e7c1f33-f809-4677-b63d-1d8ad1a36551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 29 23:01:55.579383 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:55.579345 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 29 23:01:55.579563 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:55.579414 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" podUID="4e7c1f33-f809-4677-b63d-1d8ad1a36551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 29 23:01:56.579378 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:56.579342 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 29 23:01:56.579840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:56.579403 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" podUID="4e7c1f33-f809-4677-b63d-1d8ad1a36551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 29 23:01:57.584209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:57.582901 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 29 23:01:57.584209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:57.582972 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" podUID="4e7c1f33-f809-4677-b63d-1d8ad1a36551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 29 23:01:57.906444 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:57.906367 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk"] Oct 29 23:01:57.911645 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:57.911621 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" Oct 29 23:01:57.948984 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:57.943010 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-50734e-serving-cert\"" Oct 29 23:01:57.953311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:57.953272 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/078c4461-ea44-4c8c-af2e-ae1e44776e79-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk\" (UID: \"078c4461-ea44-4c8c-af2e-ae1e44776e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" Oct 29 23:01:57.953641 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:57.953620 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078c4461-ea44-4c8c-af2e-ae1e44776e79-metrics-certs\") pod \"maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk\" (UID: \"078c4461-ea44-4c8c-af2e-ae1e44776e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" Oct 29 23:01:57.953902 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:57.953885 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078c4461-ea44-4c8c-af2e-ae1e44776e79-maas-api-tls\") pod \"maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk\" (UID: \"078c4461-ea44-4c8c-af2e-ae1e44776e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" Oct 29 23:01:57.954164 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:57.954148 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqv4f\" (UniqueName: \"kubernetes.io/projected/078c4461-ea44-4c8c-af2e-ae1e44776e79-kube-api-access-zqv4f\") pod \"maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk\" (UID: \"078c4461-ea44-4c8c-af2e-ae1e44776e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" Oct 29 23:01:57.966084 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:57.966057 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk"] Oct 29 23:01:58.056242 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:58.056175 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqv4f\" (UniqueName: \"kubernetes.io/projected/078c4461-ea44-4c8c-af2e-ae1e44776e79-kube-api-access-zqv4f\") pod \"maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk\" (UID: \"078c4461-ea44-4c8c-af2e-ae1e44776e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" Oct 29 23:01:58.056509 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:58.056416 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/078c4461-ea44-4c8c-af2e-ae1e44776e79-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk\" (UID: \"078c4461-ea44-4c8c-af2e-ae1e44776e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" Oct 29 23:01:58.056509 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:58.056466 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078c4461-ea44-4c8c-af2e-ae1e44776e79-metrics-certs\") pod \"maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk\" (UID: \"078c4461-ea44-4c8c-af2e-ae1e44776e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" Oct 29 23:01:58.056650 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:58.056535 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078c4461-ea44-4c8c-af2e-ae1e44776e79-maas-api-tls\") pod \"maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk\" (UID: \"078c4461-ea44-4c8c-af2e-ae1e44776e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" Oct 29 23:01:58.056775 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:01:58.056743 5057 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-serving-cert: secret "maas-api-e2e-ipp-b-50734e-serving-cert" not found Oct 29 23:01:58.056871 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:01:58.056825 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078c4461-ea44-4c8c-af2e-ae1e44776e79-maas-api-tls podName:078c4461-ea44-4c8c-af2e-ae1e44776e79 nodeName:}" failed. No retries permitted until 2026-10-29 23:01:58.556803336 +0000 UTC m=+1165.208301844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/078c4461-ea44-4c8c-af2e-ae1e44776e79-maas-api-tls") pod "maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" (UID: "078c4461-ea44-4c8c-af2e-ae1e44776e79") : secret "maas-api-e2e-ipp-b-50734e-serving-cert" not found Oct 29 23:01:58.057807 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:58.057777 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/078c4461-ea44-4c8c-af2e-ae1e44776e79-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk\" (UID: \"078c4461-ea44-4c8c-af2e-ae1e44776e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" Oct 29 23:01:58.062384 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:58.062358 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078c4461-ea44-4c8c-af2e-ae1e44776e79-metrics-certs\") pod \"maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk\" (UID: \"078c4461-ea44-4c8c-af2e-ae1e44776e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" Oct 29 23:01:58.081753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:58.081727 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqv4f\" (UniqueName: \"kubernetes.io/projected/078c4461-ea44-4c8c-af2e-ae1e44776e79-kube-api-access-zqv4f\") pod \"maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk\" (UID: \"078c4461-ea44-4c8c-af2e-ae1e44776e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" Oct 29 23:01:58.579917 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:58.579874 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 29 23:01:58.580133 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:58.579934 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" podUID="4e7c1f33-f809-4677-b63d-1d8ad1a36551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 29 23:01:58.585494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:58.585459 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078c4461-ea44-4c8c-af2e-ae1e44776e79-maas-api-tls\") pod \"maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk\" (UID: \"078c4461-ea44-4c8c-af2e-ae1e44776e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" Oct 29 23:01:58.588361 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:58.588322 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078c4461-ea44-4c8c-af2e-ae1e44776e79-maas-api-tls\") pod \"maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk\" (UID: \"078c4461-ea44-4c8c-af2e-ae1e44776e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" Oct 29 23:01:58.867051 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:58.866553 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" Oct 29 23:01:59.413433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:59.405261 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk"] Oct 29 23:01:59.672400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:59.601285 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 29 23:01:59.672400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:59.601383 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" podUID="4e7c1f33-f809-4677-b63d-1d8ad1a36551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 29 23:01:59.761245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:01:59.758134 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" event={"ID":"078c4461-ea44-4c8c-af2e-ae1e44776e79","Type":"ContainerStarted","Data":"9a4a253c9b7e53d7863a5120688c292821cd87a8de0f56cdf8f9c167b77161d8"} Oct 29 23:02:00.466742 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:00.466711 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd" Oct 29 23:02:00.467168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:00.467148 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr" Oct 29 23:02:00.587850 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:00.587818 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:02:00.588680 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:00.588658 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:02:00.590422 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:00.590400 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:02:00.961940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:00.961900 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" event={"ID":"078c4461-ea44-4c8c-af2e-ae1e44776e79","Type":"ContainerStarted","Data":"f24471f62e56fafc44e36717f9a7073e9af37f5523d1ffdf0fa760652ef16357"} Oct 29 23:02:00.962424 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:00.961968 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" Oct 29 23:02:01.015445 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:01.015381 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" podStartSLOduration=3.482922234 podStartE2EDuration="4.015360485s" totalImagesPullingTime="532.438251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:01.007401313 +0000 UTC m=+1167.658899862" watchObservedRunningTime="2026-10-29 23:02:01.015360485 +0000 UTC m=+1167.666859017" Oct 29 23:02:03.920364 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:02:03.920328 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:02:06.977826 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:06.977795 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" Oct 29 23:02:08.688753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:08.688719 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx"] Oct 29 23:02:08.694988 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:08.694967 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx" Oct 29 23:02:08.717095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:08.717065 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx"] Oct 29 23:02:08.725376 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:08.725348 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d430df-dockercfg-qwkfl\"" Oct 29 23:02:08.725661 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:08.725631 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d430df\"" Oct 29 23:02:08.730065 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:08.730018 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sfrw\" (UniqueName: \"kubernetes.io/projected/79d001db-e9ee-46aa-9475-5306f125d606-kube-api-access-8sfrw\") pod \"payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx\" (UID: \"79d001db-e9ee-46aa-9475-5306f125d606\") " pod="openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx" Oct 29 23:02:08.730406 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:08.730378 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d001db-e9ee-46aa-9475-5306f125d606-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx\" (UID: \"79d001db-e9ee-46aa-9475-5306f125d606\") " pod="openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx" Oct 29 23:02:08.754835 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:08.754802 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl"] Oct 29 23:02:08.770932 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:08.770906 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl" Oct 29 23:02:08.771685 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:08.771616 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl"] Oct 29 23:02:08.850212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:08.834657 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sfrw\" (UniqueName: \"kubernetes.io/projected/79d001db-e9ee-46aa-9475-5306f125d606-kube-api-access-8sfrw\") pod \"payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx\" (UID: \"79d001db-e9ee-46aa-9475-5306f125d606\") " pod="openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx" Oct 29 23:02:08.850212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:08.834778 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d001db-e9ee-46aa-9475-5306f125d606-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx\" (UID: \"79d001db-e9ee-46aa-9475-5306f125d606\") " pod="openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx" Oct 29 23:02:08.850212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:08.834856 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e0999a-213d-4f93-bbd4-3c007ae1d7eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl\" (UID: \"16e0999a-213d-4f93-bbd4-3c007ae1d7eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl" Oct 29 23:02:08.850212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:08.835046 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qjfg\" (UniqueName: \"kubernetes.io/projected/16e0999a-213d-4f93-bbd4-3c007ae1d7eb-kube-api-access-5qjfg\") pod \"payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl\" (UID: \"16e0999a-213d-4f93-bbd4-3c007ae1d7eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl" Oct 29 23:02:08.850212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:08.836554 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d001db-e9ee-46aa-9475-5306f125d606-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx\" (UID: \"79d001db-e9ee-46aa-9475-5306f125d606\") " pod="openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx" Oct 29 23:02:08.995268 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:08.993159 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e0999a-213d-4f93-bbd4-3c007ae1d7eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl\" (UID: \"16e0999a-213d-4f93-bbd4-3c007ae1d7eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl" Oct 29 23:02:09.040213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:09.014274 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e0999a-213d-4f93-bbd4-3c007ae1d7eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl\" (UID: \"16e0999a-213d-4f93-bbd4-3c007ae1d7eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl" Oct 29 23:02:09.040213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:09.014656 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qjfg\" (UniqueName: \"kubernetes.io/projected/16e0999a-213d-4f93-bbd4-3c007ae1d7eb-kube-api-access-5qjfg\") pod \"payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl\" (UID: \"16e0999a-213d-4f93-bbd4-3c007ae1d7eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl" Oct 29 23:02:09.174210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:09.146955 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qjfg\" (UniqueName: \"kubernetes.io/projected/16e0999a-213d-4f93-bbd4-3c007ae1d7eb-kube-api-access-5qjfg\") pod \"payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl\" (UID: \"16e0999a-213d-4f93-bbd4-3c007ae1d7eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl" Oct 29 23:02:09.228619 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:09.223553 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sfrw\" (UniqueName: \"kubernetes.io/projected/79d001db-e9ee-46aa-9475-5306f125d606-kube-api-access-8sfrw\") pod \"payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx\" (UID: \"79d001db-e9ee-46aa-9475-5306f125d606\") " pod="openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx" Oct 29 23:02:09.441600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:09.441569 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx" Oct 29 23:02:09.443595 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:09.443232 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl" Oct 29 23:02:09.777985 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:09.777831 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx"] Oct 29 23:02:09.798483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:09.798456 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl"] Oct 29 23:02:09.799317 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:02:09.799293 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e0999a_213d_4f93_bbd4_3c007ae1d7eb.slice/crio-3811878037b1d6a4cbb8854d0be422b7238080ff6bc0a456e771e5d67bfc80ee WatchSource:0}: Error finding container 3811878037b1d6a4cbb8854d0be422b7238080ff6bc0a456e771e5d67bfc80ee: Status 404 returned error can't find the container with id 3811878037b1d6a4cbb8854d0be422b7238080ff6bc0a456e771e5d67bfc80ee Oct 29 23:02:10.454496 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:10.454460 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl" event={"ID":"16e0999a-213d-4f93-bbd4-3c007ae1d7eb","Type":"ContainerStarted","Data":"b6fe955af1f3c1e1ad4b3b8f8246fec2e82294a936ffe75f0c529191a267b446"} Oct 29 23:02:10.454706 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:10.454526 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl" event={"ID":"16e0999a-213d-4f93-bbd4-3c007ae1d7eb","Type":"ContainerStarted","Data":"3811878037b1d6a4cbb8854d0be422b7238080ff6bc0a456e771e5d67bfc80ee"} Oct 29 23:02:10.454706 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:10.454553 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl" Oct 29 23:02:10.457343 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:10.457312 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx" event={"ID":"79d001db-e9ee-46aa-9475-5306f125d606","Type":"ContainerStarted","Data":"fffee417c15d544fcec34526b583c155c2bcf8c585f74772b8f5de9c99eae87a"} Oct 29 23:02:10.457508 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:10.457374 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx" event={"ID":"79d001db-e9ee-46aa-9475-5306f125d606","Type":"ContainerStarted","Data":"26d8a8f40d32ee4fecb9f15db1540f51e2c6513904ca345b669d169092599e42"} Oct 29 23:02:10.457508 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:10.457406 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx" Oct 29 23:02:10.504069 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:10.504020 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl" podStartSLOduration=2.504006001 podStartE2EDuration="2.504006001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:10.498378885 +0000 UTC m=+1177.149877415" watchObservedRunningTime="2026-10-29 23:02:10.504006001 +0000 UTC m=+1177.155504530" Oct 29 23:02:10.549410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:10.549349 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx" podStartSLOduration=2.549327583 podStartE2EDuration="2.549327583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:10.542119236 +0000 UTC m=+1177.193617766" watchObservedRunningTime="2026-10-29 23:02:10.549327583 +0000 UTC m=+1177.200826114" Oct 29 23:02:11.114222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:11.114170 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np"] Oct 29 23:02:11.152703 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:11.152666 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q"] Oct 29 23:02:16.115241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.115168 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np" podUID="660ff606-64f9-4a49-a739-7d0452742abf" containerName="payload-processing" containerID="cri-o://1c368543f61f80c59779028be727a2c9f423b0984f9ed9885052f3a87528e462" gracePeriod=25 Oct 29 23:02:16.157211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.153285 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q" podUID="b774656b-b5c6-4aae-bfa4-e269ced7a278" containerName="payload-pre-processing" containerID="cri-o://5875e6006f3b9fd69d2f19e204c0a45bcaac4d343d195830322f90bd0ba855f3" gracePeriod=25 Oct 29 23:02:16.567010 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.566742 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np" Oct 29 23:02:16.579109 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.579076 5057 generic.go:356] "Generic (PLEG): container finished" podID="660ff606-64f9-4a49-a739-7d0452742abf" containerID="1c368543f61f80c59779028be727a2c9f423b0984f9ed9885052f3a87528e462" exitCode=0 Oct 29 23:02:16.579256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.579124 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np" Oct 29 23:02:16.579256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.579155 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np" event={"ID":"660ff606-64f9-4a49-a739-7d0452742abf","Type":"ContainerDied","Data":"1c368543f61f80c59779028be727a2c9f423b0984f9ed9885052f3a87528e462"} Oct 29 23:02:16.579256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.579236 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np" event={"ID":"660ff606-64f9-4a49-a739-7d0452742abf","Type":"ContainerDied","Data":"a5efb7bc98ae168deea2059343fa482699d8b2082f761eaf0b58c2f63c3497f1"} Oct 29 23:02:16.579445 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.579266 5057 scope.go:122] "RemoveContainer" containerID="1c368543f61f80c59779028be727a2c9f423b0984f9ed9885052f3a87528e462" Oct 29 23:02:16.596471 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.596443 5057 generic.go:356] "Generic (PLEG): container finished" podID="b774656b-b5c6-4aae-bfa4-e269ced7a278" containerID="5875e6006f3b9fd69d2f19e204c0a45bcaac4d343d195830322f90bd0ba855f3" exitCode=0 Oct 29 23:02:16.596600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.596504 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q" event={"ID":"b774656b-b5c6-4aae-bfa4-e269ced7a278","Type":"ContainerDied","Data":"5875e6006f3b9fd69d2f19e204c0a45bcaac4d343d195830322f90bd0ba855f3"} Oct 29 23:02:16.600156 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.600136 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q" Oct 29 23:02:16.601678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.601660 5057 scope.go:122] "RemoveContainer" containerID="1c368543f61f80c59779028be727a2c9f423b0984f9ed9885052f3a87528e462" Oct 29 23:02:16.601933 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:02:16.601914 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c368543f61f80c59779028be727a2c9f423b0984f9ed9885052f3a87528e462\": container with ID starting with 1c368543f61f80c59779028be727a2c9f423b0984f9ed9885052f3a87528e462 not found: ID does not exist" containerID="1c368543f61f80c59779028be727a2c9f423b0984f9ed9885052f3a87528e462" Oct 29 23:02:16.601975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.601941 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c368543f61f80c59779028be727a2c9f423b0984f9ed9885052f3a87528e462"} err="failed to get container status \"1c368543f61f80c59779028be727a2c9f423b0984f9ed9885052f3a87528e462\": rpc error: code = NotFound desc = could not find container \"1c368543f61f80c59779028be727a2c9f423b0984f9ed9885052f3a87528e462\": container with ID starting with 1c368543f61f80c59779028be727a2c9f423b0984f9ed9885052f3a87528e462 not found: ID does not exist" Oct 29 23:02:16.631164 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.631132 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660ff606-64f9-4a49-a739-7d0452742abf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660ff606-64f9-4a49-a739-7d0452742abf-plugins-config-volume\") pod \"660ff606-64f9-4a49-a739-7d0452742abf\" (UID: \"660ff606-64f9-4a49-a739-7d0452742abf\") " Oct 29 23:02:16.631373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.631345 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660ff606-64f9-4a49-a739-7d0452742abf-kube-api-access-dh9hg\" (UniqueName: \"kubernetes.io/projected/660ff606-64f9-4a49-a739-7d0452742abf-kube-api-access-dh9hg\") pod \"660ff606-64f9-4a49-a739-7d0452742abf\" (UID: \"660ff606-64f9-4a49-a739-7d0452742abf\") " Oct 29 23:02:16.631699 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.631674 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660ff606-64f9-4a49-a739-7d0452742abf-plugins-config-volume" pod "660ff606-64f9-4a49-a739-7d0452742abf" (UID: "660ff606-64f9-4a49-a739-7d0452742abf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:16.632330 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.632307 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660ff606-64f9-4a49-a739-7d0452742abf-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:02:16.634468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.634445 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660ff606-64f9-4a49-a739-7d0452742abf-kube-api-access-dh9hg" pod "660ff606-64f9-4a49-a739-7d0452742abf" (UID: "660ff606-64f9-4a49-a739-7d0452742abf"). InnerVolumeSpecName "kube-api-access-dh9hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:16.735012 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.734965 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b774656b-b5c6-4aae-bfa4-e269ced7a278-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b774656b-b5c6-4aae-bfa4-e269ced7a278-plugins-config-volume\") pod \"b774656b-b5c6-4aae-bfa4-e269ced7a278\" (UID: \"b774656b-b5c6-4aae-bfa4-e269ced7a278\") " Oct 29 23:02:16.735275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.735255 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b774656b-b5c6-4aae-bfa4-e269ced7a278-kube-api-access-sjf4m\" (UniqueName: \"kubernetes.io/projected/b774656b-b5c6-4aae-bfa4-e269ced7a278-kube-api-access-sjf4m\") pod \"b774656b-b5c6-4aae-bfa4-e269ced7a278\" (UID: \"b774656b-b5c6-4aae-bfa4-e269ced7a278\") " Oct 29 23:02:16.735741 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.735461 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b774656b-b5c6-4aae-bfa4-e269ced7a278-plugins-config-volume" pod "b774656b-b5c6-4aae-bfa4-e269ced7a278" (UID: "b774656b-b5c6-4aae-bfa4-e269ced7a278"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:16.736235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.736213 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b774656b-b5c6-4aae-bfa4-e269ced7a278-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:02:16.736353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.736240 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh9hg\" (UniqueName: \"kubernetes.io/projected/660ff606-64f9-4a49-a739-7d0452742abf-kube-api-access-dh9hg\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:02:16.738050 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.738023 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b774656b-b5c6-4aae-bfa4-e269ced7a278-kube-api-access-sjf4m" pod "b774656b-b5c6-4aae-bfa4-e269ced7a278" (UID: "b774656b-b5c6-4aae-bfa4-e269ced7a278"). InnerVolumeSpecName "kube-api-access-sjf4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:16.838859 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.838814 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjf4m\" (UniqueName: \"kubernetes.io/projected/b774656b-b5c6-4aae-bfa4-e269ced7a278-kube-api-access-sjf4m\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:02:16.931309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.931271 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np"] Oct 29 23:02:16.935833 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:16.935804 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79e6c631-7c9fd55b5b-6v5np"] Oct 29 23:02:17.487787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:17.487751 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660ff606-64f9-4a49-a739-7d0452742abf" path="/var/lib/kubelet/pods/660ff606-64f9-4a49-a739-7d0452742abf/volumes" Oct 29 23:02:17.606050 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:17.606025 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q" Oct 29 23:02:17.606296 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:17.606136 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q" event={"ID":"b774656b-b5c6-4aae-bfa4-e269ced7a278","Type":"ContainerDied","Data":"d82c3ad427dfb87ee1247e6c36f39a154b526a50a8a6e7e10c7af7a2efea0337"} Oct 29 23:02:17.606296 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:17.606236 5057 scope.go:122] "RemoveContainer" containerID="5875e6006f3b9fd69d2f19e204c0a45bcaac4d343d195830322f90bd0ba855f3" Oct 29 23:02:17.666690 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:17.666656 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q"] Oct 29 23:02:17.680296 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:17.680267 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-lvq8q"] Oct 29 23:02:19.485476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:19.485442 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b774656b-b5c6-4aae-bfa4-e269ced7a278" path="/var/lib/kubelet/pods/b774656b-b5c6-4aae-bfa4-e269ced7a278/volumes" Oct 29 23:02:20.867355 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:20.867318 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:02:21.499677 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:21.499650 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl" Oct 29 23:02:21.501509 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:21.501488 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx" Oct 29 23:02:27.054347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:27.054296 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:02:28.816278 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:28.815276 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 23:02:28.925314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:28.925272 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 23:02:29.465232 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:29.456608 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 23:02:29.941275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:29.938348 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:02:29.942647 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:29.942613 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:02:30.089210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:30.048008 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:02:30.414333 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:30.409135 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:02:30.474207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:30.473367 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:02:30.505208 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:02:30.498399 5057 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.024s" Oct 29 23:02:30.505208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:30.498586 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:02:30.505208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:30.498628 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:02:30.505208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:30.498662 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:02:31.006710 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:30.985785 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:02:31.006710 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:30.987255 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:02:31.087663 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:31.087602 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:02:31.089607 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:31.089577 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:02:31.090355 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:31.090337 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:02:31.715580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:31.638281 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:02:31.988514 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:31.988419 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:02:32.149207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:32.121003 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 23:02:32.785249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:32.759100 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:02:32.883225 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:02:32.882597 5057 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.257s" Oct 29 23:02:32.883225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:32.882681 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:02:32.883225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:32.882741 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:02:32.995682 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:32.995637 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:02:32.999713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:32.996447 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:02:33.116491 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:33.116392 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:02:33.292561 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:33.292519 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 23:02:33.293621 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:33.293586 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:02:33.367305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:33.366942 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:02:33.391933 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:33.391854 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:02:33.973701 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:02:33.973478 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:02:34.695626 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:34.695593 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:02:34.701821 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:34.701795 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:02:34.788660 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:34.788304 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:02:34.788660 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:34.788383 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" podUID="7cf5e014-85ba-4024-9ebc-088735098c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:02:37.960686 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:37.960329 5057 patch_prober.go:29] interesting pod/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:02:37.960686 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:37.960399 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" podUID="b39228ac-4129-460c-9a6c-42efc3a207bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:02:39.128338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:39.128304 5057 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:02:39.128849 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:39.128431 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" podUID="9864c587-073a-4b1e-9e7c-68fe058b3c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:02:40.898345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:40.898316 5057 generic.go:356] "Generic (PLEG): container finished" podID="9864c587-073a-4b1e-9e7c-68fe058b3c16" containerID="9603457875dc61f1e283129cf13a3ff129ec11b88905bc93ac83fe06f50fcf27" exitCode=0 Oct 29 23:02:40.898830 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:40.898351 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" event={"ID":"9864c587-073a-4b1e-9e7c-68fe058b3c16","Type":"ContainerDied","Data":"9603457875dc61f1e283129cf13a3ff129ec11b88905bc93ac83fe06f50fcf27"} Oct 29 23:02:40.898900 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:40.898872 5057 scope.go:122] "RemoveContainer" containerID="9603457875dc61f1e283129cf13a3ff129ec11b88905bc93ac83fe06f50fcf27" Oct 29 23:02:40.900772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:40.900740 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:02:40.900859 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:40.900813 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:02:40.900859 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:40.900847 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:02:42.251052 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:42.251013 5057 generic.go:356] "Generic (PLEG): container finished" podID="7cf5e014-85ba-4024-9ebc-088735098c5c" containerID="834443d9a86fb7261754ef952239745c80d8ff100b89cffcbebc26741059c880" exitCode=0 Oct 29 23:02:42.253417 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:42.253380 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" event={"ID":"7cf5e014-85ba-4024-9ebc-088735098c5c","Type":"ContainerDied","Data":"834443d9a86fb7261754ef952239745c80d8ff100b89cffcbebc26741059c880"} Oct 29 23:02:42.254520 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:42.254353 5057 scope.go:122] "RemoveContainer" containerID="834443d9a86fb7261754ef952239745c80d8ff100b89cffcbebc26741059c880" Oct 29 23:02:42.260105 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:42.260041 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:02:42.260931 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:42.260883 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:02:42.261169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:42.261142 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:02:42.369705 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:42.369618 5057 generic.go:356] "Generic (PLEG): container finished" podID="b39228ac-4129-460c-9a6c-42efc3a207bb" containerID="b76a1dead114db83f75eb9643df5f0ed2b3bcf297a1746b64661dd1ebfe04460" exitCode=0 Oct 29 23:02:42.369894 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:42.369857 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" event={"ID":"b39228ac-4129-460c-9a6c-42efc3a207bb","Type":"ContainerDied","Data":"b76a1dead114db83f75eb9643df5f0ed2b3bcf297a1746b64661dd1ebfe04460"} Oct 29 23:02:42.370653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:42.370584 5057 scope.go:122] "RemoveContainer" containerID="b76a1dead114db83f75eb9643df5f0ed2b3bcf297a1746b64661dd1ebfe04460" Oct 29 23:02:42.382682 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:42.382653 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" event={"ID":"9864c587-073a-4b1e-9e7c-68fe058b3c16","Type":"ContainerStarted","Data":"d300568f26076396d8067fcd3f755a541b30555047ed82d19994391a1e43440a"} Oct 29 23:02:42.403124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:42.392781 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:02:42.403124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:42.392955 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:02:42.403124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:42.393065 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:02:42.860267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:42.842141 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:02:42.861464 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:42.845283 5057 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 29 23:02:42.861712 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:42.861682 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" podUID="9864c587-073a-4b1e-9e7c-68fe058b3c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 29 23:02:43.803212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:43.800257 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" event={"ID":"7cf5e014-85ba-4024-9ebc-088735098c5c","Type":"ContainerStarted","Data":"d53ce11747772ebcbc2daaefe98ce68370a0c18afac031ad86a492a97d69dee9"} Oct 29 23:02:43.803212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:43.802410 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" event={"ID":"b39228ac-4129-460c-9a6c-42efc3a207bb","Type":"ContainerStarted","Data":"f87a21640e31ee1117dc1d85d1c04f7b10659a2fe5ea0d3975bbec5d19c8e402"} Oct 29 23:02:43.849141 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:43.848508 5057 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 29 23:02:43.849141 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:43.848569 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" podUID="9864c587-073a-4b1e-9e7c-68fe058b3c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 29 23:02:43.963054 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:43.961316 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:02:43.963054 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:43.962456 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:02:43.963054 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:43.962517 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" podUID="7cf5e014-85ba-4024-9ebc-088735098c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 29 23:02:44.129264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:44.114278 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:02:44.129264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:44.114404 5057 patch_prober.go:29] interesting pod/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 23:02:44.129264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:44.114453 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" podUID="b39228ac-4129-460c-9a6c-42efc3a207bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 23:02:44.862211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:44.854467 5057 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 29 23:02:44.862211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:44.854527 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" podUID="9864c587-073a-4b1e-9e7c-68fe058b3c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 29 23:02:44.954354 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:44.953891 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:02:44.954354 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:44.953957 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" podUID="7cf5e014-85ba-4024-9ebc-088735098c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 29 23:02:45.109267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:45.107570 5057 patch_prober.go:29] interesting pod/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 23:02:45.109267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:45.107634 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" podUID="b39228ac-4129-460c-9a6c-42efc3a207bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 23:02:45.851211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:45.846525 5057 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 29 23:02:45.851211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:45.846597 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" podUID="9864c587-073a-4b1e-9e7c-68fe058b3c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 29 23:02:45.950393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:45.950352 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:02:45.951006 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:45.950972 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" podUID="7cf5e014-85ba-4024-9ebc-088735098c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 29 23:02:46.108568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:46.108446 5057 patch_prober.go:29] interesting pod/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 23:02:46.136324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:46.136276 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" podUID="b39228ac-4129-460c-9a6c-42efc3a207bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 23:02:46.974213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:46.934602 5057 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 29 23:02:46.974213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:46.934661 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" podUID="9864c587-073a-4b1e-9e7c-68fe058b3c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 29 23:02:47.082208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:47.070790 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:02:47.082208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:47.070851 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" podUID="7cf5e014-85ba-4024-9ebc-088735098c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 29 23:02:47.111207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:47.109858 5057 patch_prober.go:29] interesting pod/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 23:02:47.111207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:47.109913 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" podUID="b39228ac-4129-460c-9a6c-42efc3a207bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 23:02:47.901206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:47.894753 5057 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 29 23:02:47.901206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:47.894815 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" podUID="9864c587-073a-4b1e-9e7c-68fe058b3c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 29 23:02:48.025210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:47.955872 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:02:48.025210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:47.988086 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" podUID="7cf5e014-85ba-4024-9ebc-088735098c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 29 23:02:48.117989 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:48.117954 5057 patch_prober.go:29] interesting pod/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 23:02:48.118263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:48.118239 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" podUID="b39228ac-4129-460c-9a6c-42efc3a207bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 23:02:48.794223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:48.790742 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:02:49.030643 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:49.030593 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:02:49.067257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:49.043302 5057 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 29 23:02:49.067257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:49.043368 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" podUID="9864c587-073a-4b1e-9e7c-68fe058b3c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 29 23:02:49.067257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:49.043831 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:02:49.067257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:49.043864 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" podUID="7cf5e014-85ba-4024-9ebc-088735098c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 29 23:02:49.110210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:49.110062 5057 patch_prober.go:29] interesting pod/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 23:02:49.110210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:49.110120 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" podUID="b39228ac-4129-460c-9a6c-42efc3a207bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 23:02:49.218179 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:49.218127 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:02:49.739569 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:49.739528 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:02:49.852323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:49.851328 5057 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 29 23:02:49.852323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:49.851386 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" podUID="9864c587-073a-4b1e-9e7c-68fe058b3c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 29 23:02:49.952291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:49.952252 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:02:49.952581 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:49.952541 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" podUID="7cf5e014-85ba-4024-9ebc-088735098c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 29 23:02:50.031509 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:50.031419 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:02:50.032555 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:50.032533 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:02:50.147211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:50.113938 5057 patch_prober.go:29] interesting pod/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 23:02:50.147211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:50.113997 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" podUID="b39228ac-4129-460c-9a6c-42efc3a207bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 23:02:50.243245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:50.231849 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 23:02:50.243245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:50.232511 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:02:50.464220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:50.450751 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded" Oct 29 23:02:50.757267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:50.754308 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:02:50.757267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:50.755030 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:02:50.917004 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:50.896224 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:02:50.917742 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:50.917712 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:02:50.920338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:50.918308 5057 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 29 23:02:50.920338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:50.918371 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" podUID="9864c587-073a-4b1e-9e7c-68fe058b3c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 29 23:02:50.948257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:50.948219 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:02:50.948525 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:50.948497 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" podUID="7cf5e014-85ba-4024-9ebc-088735098c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 29 23:02:51.102587 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:51.102487 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:02:51.108403 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:51.108334 5057 patch_prober.go:29] interesting pod/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 23:02:51.108671 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:51.108645 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" podUID="b39228ac-4129-460c-9a6c-42efc3a207bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 23:02:51.889310 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:51.889258 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:02:51.889835 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:51.889809 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:02:51.890885 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:51.890860 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:02:51.918395 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:51.918363 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:02:51.970008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:51.956837 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:02:51.970008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:51.956897 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" podUID="7cf5e014-85ba-4024-9ebc-088735098c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 29 23:02:52.106486 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:52.106456 5057 patch_prober.go:29] interesting pod/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 23:02:52.107050 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:52.107020 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" podUID="b39228ac-4129-460c-9a6c-42efc3a207bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 23:02:52.989812 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:52.989777 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:02:52.992013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:52.991986 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:02:53.354664 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:53.354584 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:02:53.364149 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:02:53.364120 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:03:03.988457 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:03:03.988384 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:03:10.915627 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:10.915584 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8"] Oct 29 23:03:10.916130 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:10.915869 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" podUID="51473c56-3874-4c1b-af5b-0601a57b1d8f" containerName="istio-proxy" containerID="cri-o://e1701cd0fa6ef8f147d7dbf57b93387a4aff3aefb4f9db27af8605dc0c667788" gracePeriod=30 Oct 29 23:03:11.425100 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:11.425065 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:03:11.711430 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:11.711347 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr"] Oct 29 23:03:11.812325 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:11.812256 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd"] Oct 29 23:03:14.192239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:14.190526 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 23:03:14.588285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:14.588083 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6"] Oct 29 23:03:14.601347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:14.588401 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" podUID="e0c58b99-cc75-42a7-8ba1-8734214ca340" containerName="istio-proxy" containerID="cri-o://2fb7a38252a53a5b475c35550ff5beb9cdbbd5fdb1707f81621892c0d3f0800b" gracePeriod=30 Oct 29 23:03:14.915605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:14.915542 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:15.026698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:15.026657 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj"] Oct 29 23:03:15.051657 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:15.051603 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" podUID="ed5d8696-66e9-4e1d-9419-2597fef50ec7" containerName="istio-proxy" containerID="cri-o://5bbef6450b52a2881cecb2cd8432e4ef841f31ac3722466986bc1f46107a55bf" gracePeriod=30 Oct 29 23:03:15.094402 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:15.094351 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:15.188896 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:15.153237 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7"] Oct 29 23:03:15.188896 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:15.153499 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" podUID="d205536f-2f06-4f8a-9ef1-e640f0904f6c" containerName="istio-proxy" containerID="cri-o://565e7676cf8de179628f79102d9f0f0b9fda2fbeac48cfcb067a103aef2dc4ce" gracePeriod=30 Oct 29 23:03:16.070052 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.070017 5057 generic.go:356] "Generic (PLEG): container finished" podID="51473c56-3874-4c1b-af5b-0601a57b1d8f" containerID="e1701cd0fa6ef8f147d7dbf57b93387a4aff3aefb4f9db27af8605dc0c667788" exitCode=0 Oct 29 23:03:16.070466 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.070113 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" event={"ID":"51473c56-3874-4c1b-af5b-0601a57b1d8f","Type":"ContainerDied","Data":"e1701cd0fa6ef8f147d7dbf57b93387a4aff3aefb4f9db27af8605dc0c667788"} Oct 29 23:03:16.365849 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.365457 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 23:03:16.483281 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.483247 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51473c56-3874-4c1b-af5b-0601a57b1d8f-kube-api-access-545dw\" (UniqueName: \"kubernetes.io/projected/51473c56-3874-4c1b-af5b-0601a57b1d8f-kube-api-access-545dw\") pod \"51473c56-3874-4c1b-af5b-0601a57b1d8f\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " Oct 29 23:03:16.483484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.483316 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-envoy\") pod \"51473c56-3874-4c1b-af5b-0601a57b1d8f\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " Oct 29 23:03:16.483484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.483389 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-podinfo\") pod \"51473c56-3874-4c1b-af5b-0601a57b1d8f\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " Oct 29 23:03:16.483484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.483415 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-token\" (UniqueName: \"kubernetes.io/projected/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-token\") pod \"51473c56-3874-4c1b-af5b-0601a57b1d8f\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " Oct 29 23:03:16.483653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.483506 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-workload-socket\") pod \"51473c56-3874-4c1b-af5b-0601a57b1d8f\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " Oct 29 23:03:16.483653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.483572 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-workload-certs\") pod \"51473c56-3874-4c1b-af5b-0601a57b1d8f\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " Oct 29 23:03:16.483794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.483779 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-data\") pod \"51473c56-3874-4c1b-af5b-0601a57b1d8f\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " Oct 29 23:03:16.483878 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.483837 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-credential-socket\") pod \"51473c56-3874-4c1b-af5b-0601a57b1d8f\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " Oct 29 23:03:16.483878 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.483866 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51473c56-3874-4c1b-af5b-0601a57b1d8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51473c56-3874-4c1b-af5b-0601a57b1d8f-istiod-ca-cert\") pod \"51473c56-3874-4c1b-af5b-0601a57b1d8f\" (UID: \"51473c56-3874-4c1b-af5b-0601a57b1d8f\") " Oct 29 23:03:16.484850 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.484827 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51473c56-3874-4c1b-af5b-0601a57b1d8f-istiod-ca-cert" pod "51473c56-3874-4c1b-af5b-0601a57b1d8f" (UID: "51473c56-3874-4c1b-af5b-0601a57b1d8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:16.485089 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.485072 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-workload-certs" pod "51473c56-3874-4c1b-af5b-0601a57b1d8f" (UID: "51473c56-3874-4c1b-af5b-0601a57b1d8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:16.485608 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.485583 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-credential-socket" pod "51473c56-3874-4c1b-af5b-0601a57b1d8f" (UID: "51473c56-3874-4c1b-af5b-0601a57b1d8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:16.485850 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.485812 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-data" pod "51473c56-3874-4c1b-af5b-0601a57b1d8f" (UID: "51473c56-3874-4c1b-af5b-0601a57b1d8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:16.486606 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.486584 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-workload-socket" pod "51473c56-3874-4c1b-af5b-0601a57b1d8f" (UID: "51473c56-3874-4c1b-af5b-0601a57b1d8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:16.488025 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.487997 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-envoy" pod "51473c56-3874-4c1b-af5b-0601a57b1d8f" (UID: "51473c56-3874-4c1b-af5b-0601a57b1d8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:16.488131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.488023 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-podinfo" pod "51473c56-3874-4c1b-af5b-0601a57b1d8f" (UID: "51473c56-3874-4c1b-af5b-0601a57b1d8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:03:16.488313 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.488292 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51473c56-3874-4c1b-af5b-0601a57b1d8f-kube-api-access-545dw" pod "51473c56-3874-4c1b-af5b-0601a57b1d8f" (UID: "51473c56-3874-4c1b-af5b-0601a57b1d8f"). InnerVolumeSpecName "kube-api-access-545dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:16.492944 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.492911 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-token" pod "51473c56-3874-4c1b-af5b-0601a57b1d8f" (UID: "51473c56-3874-4c1b-af5b-0601a57b1d8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:16.586973 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.586881 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:16.586973 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.586922 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:16.586973 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.586936 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:16.586973 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.586950 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:16.586973 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.586963 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:16.586973 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.586975 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:16.587462 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.586987 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51473c56-3874-4c1b-af5b-0601a57b1d8f-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:16.587462 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.587000 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-545dw\" (UniqueName: \"kubernetes.io/projected/51473c56-3874-4c1b-af5b-0601a57b1d8f-kube-api-access-545dw\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:16.587462 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.587011 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51473c56-3874-4c1b-af5b-0601a57b1d8f-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:16.712580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.712535 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr" podUID="0f0ff64c-6add-4f9c-bb35-2636838d7a10" containerName="payload-processing" containerID="cri-o://54d79c3cc3234f521b85adb3f03a11244d1eab43c6e6d4662b0cd68e34f607a0" gracePeriod=25 Oct 29 23:03:16.813438 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:16.813394 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd" podUID="af4639d7-85ad-45ba-8dda-e0b14e4bb396" containerName="payload-pre-processing" containerID="cri-o://7f5b382f7831beb2e903ab457c4c0946bc7a25e3afc51c492b04b232d00247e4" gracePeriod=25 Oct 29 23:03:17.176963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.176933 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" Oct 29 23:03:17.177506 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.177056 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8" event={"ID":"51473c56-3874-4c1b-af5b-0601a57b1d8f","Type":"ContainerDied","Data":"20b7a176fcfd4502463c8b0cb87ccc393b224f681444e221127cd968425299f6"} Oct 29 23:03:17.177506 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.177120 5057 scope.go:122] "RemoveContainer" containerID="e1701cd0fa6ef8f147d7dbf57b93387a4aff3aefb4f9db27af8605dc0c667788" Oct 29 23:03:17.182677 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.182412 5057 generic.go:356] "Generic (PLEG): container finished" podID="af4639d7-85ad-45ba-8dda-e0b14e4bb396" containerID="7f5b382f7831beb2e903ab457c4c0946bc7a25e3afc51c492b04b232d00247e4" exitCode=0 Oct 29 23:03:17.182677 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.182582 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd" event={"ID":"af4639d7-85ad-45ba-8dda-e0b14e4bb396","Type":"ContainerDied","Data":"7f5b382f7831beb2e903ab457c4c0946bc7a25e3afc51c492b04b232d00247e4"} Oct 29 23:03:17.183401 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.183378 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr" Oct 29 23:03:17.185176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.185069 5057 generic.go:356] "Generic (PLEG): container finished" podID="0f0ff64c-6add-4f9c-bb35-2636838d7a10" containerID="54d79c3cc3234f521b85adb3f03a11244d1eab43c6e6d4662b0cd68e34f607a0" exitCode=0 Oct 29 23:03:17.185176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.185128 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr" event={"ID":"0f0ff64c-6add-4f9c-bb35-2636838d7a10","Type":"ContainerDied","Data":"54d79c3cc3234f521b85adb3f03a11244d1eab43c6e6d4662b0cd68e34f607a0"} Oct 29 23:03:17.197836 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.197792 5057 scope.go:122] "RemoveContainer" containerID="54d79c3cc3234f521b85adb3f03a11244d1eab43c6e6d4662b0cd68e34f607a0" Oct 29 23:03:17.280812 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.280779 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8"] Oct 29 23:03:17.291868 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.291842 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-rwxp8"] Oct 29 23:03:17.309737 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.309705 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f0ff64c-6add-4f9c-bb35-2636838d7a10-kube-api-access-5thv8\" (UniqueName: \"kubernetes.io/projected/0f0ff64c-6add-4f9c-bb35-2636838d7a10-kube-api-access-5thv8\") pod \"0f0ff64c-6add-4f9c-bb35-2636838d7a10\" (UID: \"0f0ff64c-6add-4f9c-bb35-2636838d7a10\") " Oct 29 23:03:17.310033 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.310015 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f0ff64c-6add-4f9c-bb35-2636838d7a10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f0ff64c-6add-4f9c-bb35-2636838d7a10-plugins-config-volume\") pod \"0f0ff64c-6add-4f9c-bb35-2636838d7a10\" (UID: \"0f0ff64c-6add-4f9c-bb35-2636838d7a10\") " Oct 29 23:03:17.310481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.310458 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0ff64c-6add-4f9c-bb35-2636838d7a10-plugins-config-volume" pod "0f0ff64c-6add-4f9c-bb35-2636838d7a10" (UID: "0f0ff64c-6add-4f9c-bb35-2636838d7a10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:17.311074 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.311054 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f0ff64c-6add-4f9c-bb35-2636838d7a10-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:17.336210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.313925 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0ff64c-6add-4f9c-bb35-2636838d7a10-kube-api-access-5thv8" pod "0f0ff64c-6add-4f9c-bb35-2636838d7a10" (UID: "0f0ff64c-6add-4f9c-bb35-2636838d7a10"). InnerVolumeSpecName "kube-api-access-5thv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:17.499215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.497263 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5thv8\" (UniqueName: \"kubernetes.io/projected/0f0ff64c-6add-4f9c-bb35-2636838d7a10-kube-api-access-5thv8\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:17.613799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.601369 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51473c56-3874-4c1b-af5b-0601a57b1d8f" path="/var/lib/kubelet/pods/51473c56-3874-4c1b-af5b-0601a57b1d8f/volumes" Oct 29 23:03:17.613799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.605831 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd" Oct 29 23:03:17.715078 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.715044 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af4639d7-85ad-45ba-8dda-e0b14e4bb396-kube-api-access-r8ltw\" (UniqueName: \"kubernetes.io/projected/af4639d7-85ad-45ba-8dda-e0b14e4bb396-kube-api-access-r8ltw\") pod \"af4639d7-85ad-45ba-8dda-e0b14e4bb396\" (UID: \"af4639d7-85ad-45ba-8dda-e0b14e4bb396\") " Oct 29 23:03:17.715290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.715129 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af4639d7-85ad-45ba-8dda-e0b14e4bb396-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4639d7-85ad-45ba-8dda-e0b14e4bb396-plugins-config-volume\") pod \"af4639d7-85ad-45ba-8dda-e0b14e4bb396\" (UID: \"af4639d7-85ad-45ba-8dda-e0b14e4bb396\") " Oct 29 23:03:17.715519 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.715495 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4639d7-85ad-45ba-8dda-e0b14e4bb396-plugins-config-volume" pod "af4639d7-85ad-45ba-8dda-e0b14e4bb396" (UID: "af4639d7-85ad-45ba-8dda-e0b14e4bb396"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:17.716220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.716165 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4639d7-85ad-45ba-8dda-e0b14e4bb396-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:17.717322 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.717300 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4639d7-85ad-45ba-8dda-e0b14e4bb396-kube-api-access-r8ltw" pod "af4639d7-85ad-45ba-8dda-e0b14e4bb396" (UID: "af4639d7-85ad-45ba-8dda-e0b14e4bb396"). InnerVolumeSpecName "kube-api-access-r8ltw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:17.818361 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:17.818277 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8ltw\" (UniqueName: \"kubernetes.io/projected/af4639d7-85ad-45ba-8dda-e0b14e4bb396-kube-api-access-r8ltw\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:18.196817 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:18.196767 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd" event={"ID":"af4639d7-85ad-45ba-8dda-e0b14e4bb396","Type":"ContainerDied","Data":"41e9ef6478e80be1d00b22e90655ae77303da1c0d46e609da40d41fe416d91d9"} Oct 29 23:03:18.197339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:18.196798 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd" Oct 29 23:03:18.197453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:18.197323 5057 scope.go:122] "RemoveContainer" containerID="7f5b382f7831beb2e903ab457c4c0946bc7a25e3afc51c492b04b232d00247e4" Oct 29 23:03:18.198391 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:18.198355 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr" event={"ID":"0f0ff64c-6add-4f9c-bb35-2636838d7a10","Type":"ContainerDied","Data":"39b11d4f8dea9f4ba183b40d9b30b532745574d6c9cd87096f9ee6f0d2cd922b"} Oct 29 23:03:18.198516 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:18.198462 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr" Oct 29 23:03:18.261828 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:18.261780 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr"] Oct 29 23:03:18.272775 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:18.272745 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-67f8d75776-xmlqr"] Oct 29 23:03:18.311564 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:18.311531 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd"] Oct 29 23:03:18.314230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:18.314209 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-745dd5d8f5-clgnd"] Oct 29 23:03:19.365211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.313454 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk"] Oct 29 23:03:19.401211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.367506 5057 cpu_manager.go:414] "containerMap: removing container" podUID="0f0ff64c-6add-4f9c-bb35-2636838d7a10" containerName="payload-processing" Oct 29 23:03:19.419495 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.419465 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f0ff64c-6add-4f9c-bb35-2636838d7a10" containerName="payload-processing" Oct 29 23:03:19.517607 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.517578 5057 cpu_manager.go:414] "containerMap: removing container" podUID="51473c56-3874-4c1b-af5b-0601a57b1d8f" containerName="istio-proxy" Oct 29 23:03:19.517822 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.517805 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51473c56-3874-4c1b-af5b-0601a57b1d8f" containerName="istio-proxy" Oct 29 23:03:19.517944 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.517933 5057 cpu_manager.go:414] "containerMap: removing container" podUID="af4639d7-85ad-45ba-8dda-e0b14e4bb396" containerName="payload-pre-processing" Oct 29 23:03:19.518022 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.518012 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af4639d7-85ad-45ba-8dda-e0b14e4bb396" containerName="payload-pre-processing" Oct 29 23:03:19.518116 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.518106 5057 cpu_manager.go:414] "containerMap: removing container" podUID="660ff606-64f9-4a49-a739-7d0452742abf" containerName="payload-processing" Oct 29 23:03:19.518212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.518179 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660ff606-64f9-4a49-a739-7d0452742abf" containerName="payload-processing" Oct 29 23:03:19.518292 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.518282 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b774656b-b5c6-4aae-bfa4-e269ced7a278" containerName="payload-pre-processing" Oct 29 23:03:19.518406 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.518391 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b774656b-b5c6-4aae-bfa4-e269ced7a278" containerName="payload-pre-processing" Oct 29 23:03:19.518814 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.518792 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f0ff64c-6add-4f9c-bb35-2636838d7a10" containerName="payload-processing" Oct 29 23:03:19.518937 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.518927 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="51473c56-3874-4c1b-af5b-0601a57b1d8f" containerName="istio-proxy" Oct 29 23:03:19.519023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.519013 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="af4639d7-85ad-45ba-8dda-e0b14e4bb396" containerName="payload-pre-processing" Oct 29 23:03:19.519104 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.519095 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="660ff606-64f9-4a49-a739-7d0452742abf" containerName="payload-processing" Oct 29 23:03:19.519206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.519172 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b774656b-b5c6-4aae-bfa4-e269ced7a278" containerName="payload-pre-processing" Oct 29 23:03:19.534440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.534408 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk"] Oct 29 23:03:19.534850 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.534746 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk" Oct 29 23:03:19.557046 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.554437 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-50734e-dockercfg-782wq\"" Oct 29 23:03:19.557046 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.555142 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-50734e\"" Oct 29 23:03:19.571768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.570591 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356ee5cf-6fec-47b4-90c1-0c1354b74c18-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk\" (UID: \"356ee5cf-6fec-47b4-90c1-0c1354b74c18\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk" Oct 29 23:03:19.571768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.570656 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn2hh\" (UniqueName: \"kubernetes.io/projected/356ee5cf-6fec-47b4-90c1-0c1354b74c18-kube-api-access-jn2hh\") pod \"payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk\" (UID: \"356ee5cf-6fec-47b4-90c1-0c1354b74c18\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk" Oct 29 23:03:19.647986 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.647895 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f0ff64c-6add-4f9c-bb35-2636838d7a10" path="/var/lib/kubelet/pods/0f0ff64c-6add-4f9c-bb35-2636838d7a10/volumes" Oct 29 23:03:19.674713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.672886 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356ee5cf-6fec-47b4-90c1-0c1354b74c18-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk\" (UID: \"356ee5cf-6fec-47b4-90c1-0c1354b74c18\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk" Oct 29 23:03:19.674713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.672962 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn2hh\" (UniqueName: \"kubernetes.io/projected/356ee5cf-6fec-47b4-90c1-0c1354b74c18-kube-api-access-jn2hh\") pod \"payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk\" (UID: \"356ee5cf-6fec-47b4-90c1-0c1354b74c18\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk" Oct 29 23:03:19.674713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.674561 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356ee5cf-6fec-47b4-90c1-0c1354b74c18-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk\" (UID: \"356ee5cf-6fec-47b4-90c1-0c1354b74c18\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk" Oct 29 23:03:19.698662 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.698381 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af4639d7-85ad-45ba-8dda-e0b14e4bb396" path="/var/lib/kubelet/pods/af4639d7-85ad-45ba-8dda-e0b14e4bb396/volumes" Oct 29 23:03:19.699335 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.699309 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq"] Oct 29 23:03:19.777943 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.777908 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn2hh\" (UniqueName: \"kubernetes.io/projected/356ee5cf-6fec-47b4-90c1-0c1354b74c18-kube-api-access-jn2hh\") pod \"payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk\" (UID: \"356ee5cf-6fec-47b4-90c1-0c1354b74c18\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk" Oct 29 23:03:19.805458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.804401 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq"] Oct 29 23:03:19.805458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.804462 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx"] Oct 29 23:03:19.805458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.804592 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq" Oct 29 23:03:19.805458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.804660 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" podUID="f2b7f715-3c33-483f-9d7c-75797d057663" containerName="maas-api" containerID="cri-o://f7fe21a8787828e18c2397f8a14611827f3ac93282462d103edcdee4f1926625" gracePeriod=30 Oct 29 23:03:19.817412 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.816472 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" podUID="f2b7f715-3c33-483f-9d7c-75797d057663" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.120:8443/health\": dial tcp 10.132.0.120:8443: connect: connection refused" Oct 29 23:03:19.876465 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.874631 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk" Oct 29 23:03:19.885114 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.885078 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b49892fa-80a0-4f7b-a75d-fb027e183254-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq\" (UID: \"b49892fa-80a0-4f7b-a75d-fb027e183254\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq" Oct 29 23:03:19.886861 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:19.886832 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llxp2\" (UniqueName: \"kubernetes.io/projected/b49892fa-80a0-4f7b-a75d-fb027e183254-kube-api-access-llxp2\") pod \"payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq\" (UID: \"b49892fa-80a0-4f7b-a75d-fb027e183254\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq" Oct 29 23:03:20.167103 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.143475 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b49892fa-80a0-4f7b-a75d-fb027e183254-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq\" (UID: \"b49892fa-80a0-4f7b-a75d-fb027e183254\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq" Oct 29 23:03:20.167103 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.143754 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llxp2\" (UniqueName: \"kubernetes.io/projected/b49892fa-80a0-4f7b-a75d-fb027e183254-kube-api-access-llxp2\") pod \"payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq\" (UID: \"b49892fa-80a0-4f7b-a75d-fb027e183254\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq" Oct 29 23:03:20.167103 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.144494 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b49892fa-80a0-4f7b-a75d-fb027e183254-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq\" (UID: \"b49892fa-80a0-4f7b-a75d-fb027e183254\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq" Oct 29 23:03:20.484041 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.483909 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llxp2\" (UniqueName: \"kubernetes.io/projected/b49892fa-80a0-4f7b-a75d-fb027e183254-kube-api-access-llxp2\") pod \"payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq\" (UID: \"b49892fa-80a0-4f7b-a75d-fb027e183254\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq" Oct 29 23:03:20.590906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.589042 5057 generic.go:356] "Generic (PLEG): container finished" podID="f2b7f715-3c33-483f-9d7c-75797d057663" containerID="f7fe21a8787828e18c2397f8a14611827f3ac93282462d103edcdee4f1926625" exitCode=0 Oct 29 23:03:20.590906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.589161 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" event={"ID":"f2b7f715-3c33-483f-9d7c-75797d057663","Type":"ContainerDied","Data":"f7fe21a8787828e18c2397f8a14611827f3ac93282462d103edcdee4f1926625"} Oct 29 23:03:20.594799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.591474 5057 generic.go:356] "Generic (PLEG): container finished" podID="ed5d8696-66e9-4e1d-9419-2597fef50ec7" containerID="5bbef6450b52a2881cecb2cd8432e4ef841f31ac3722466986bc1f46107a55bf" exitCode=0 Oct 29 23:03:20.594799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.591622 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" event={"ID":"ed5d8696-66e9-4e1d-9419-2597fef50ec7","Type":"ContainerDied","Data":"5bbef6450b52a2881cecb2cd8432e4ef841f31ac3722466986bc1f46107a55bf"} Oct 29 23:03:20.594799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.593469 5057 generic.go:356] "Generic (PLEG): container finished" podID="e0c58b99-cc75-42a7-8ba1-8734214ca340" containerID="2fb7a38252a53a5b475c35550ff5beb9cdbbd5fdb1707f81621892c0d3f0800b" exitCode=0 Oct 29 23:03:20.594799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.593538 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" event={"ID":"e0c58b99-cc75-42a7-8ba1-8734214ca340","Type":"ContainerDied","Data":"2fb7a38252a53a5b475c35550ff5beb9cdbbd5fdb1707f81621892c0d3f0800b"} Oct 29 23:03:20.597492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.596729 5057 generic.go:356] "Generic (PLEG): container finished" podID="d205536f-2f06-4f8a-9ef1-e640f0904f6c" containerID="565e7676cf8de179628f79102d9f0f0b9fda2fbeac48cfcb067a103aef2dc4ce" exitCode=0 Oct 29 23:03:20.597492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.596953 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" event={"ID":"d205536f-2f06-4f8a-9ef1-e640f0904f6c","Type":"ContainerDied","Data":"565e7676cf8de179628f79102d9f0f0b9fda2fbeac48cfcb067a103aef2dc4ce"} Oct 29 23:03:20.748146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.741939 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq" Oct 29 23:03:20.764841 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.764458 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk"] Oct 29 23:03:20.894375 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.894331 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc"] Oct 29 23:03:20.986325 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.977986 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc"] Oct 29 23:03:20.986325 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.978134 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:20.998013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.997888 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsrfg\" (UniqueName: \"kubernetes.io/projected/bbfc0179-11dd-431d-af1a-0793c48ba6fe-kube-api-access-rsrfg\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:20.998013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.997986 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-credential-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:20.998013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.998012 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-envoy\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:20.998336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.998057 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-podinfo\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:20.998336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.998097 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-data\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:20.998336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.998177 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istiod-ca-cert\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:20.998336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.998255 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-workload-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:20.998336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.998289 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-token\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:20.998615 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:20.998419 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-workload-certs\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:21.006248 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.005800 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 23:03:21.126209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.123823 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsrfg\" (UniqueName: \"kubernetes.io/projected/bbfc0179-11dd-431d-af1a-0793c48ba6fe-kube-api-access-rsrfg\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:21.126209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.123920 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-credential-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:21.126209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.123947 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-envoy\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:21.126209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.123995 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-podinfo\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:21.126209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.124031 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-data\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:21.126209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.124118 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istiod-ca-cert\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:21.126209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.124168 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-workload-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:21.126209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.124214 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-token\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:21.126209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.124344 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-workload-certs\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:21.126209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.125004 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-workload-certs\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:21.185436 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.185052 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-credential-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:21.209225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.206092 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istiod-ca-cert\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:21.249373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.229911 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsrfg\" (UniqueName: \"kubernetes.io/projected/bbfc0179-11dd-431d-af1a-0793c48ba6fe-kube-api-access-rsrfg\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:21.249373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.230243 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-workload-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:21.249373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.230676 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-data\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:21.249373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.231848 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-credential-socket\") pod \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " Oct 29 23:03:21.249373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.231893 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed5d8696-66e9-4e1d-9419-2597fef50ec7-kube-api-access-r9kbv\" (UniqueName: \"kubernetes.io/projected/ed5d8696-66e9-4e1d-9419-2597fef50ec7-kube-api-access-r9kbv\") pod \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " Oct 29 23:03:21.249373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.231999 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istiod-ca-cert\") pod \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " Oct 29 23:03:21.249373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.232163 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-workload-certs\") pod \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " Oct 29 23:03:21.253158 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.251462 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istiod-ca-cert" pod "ed5d8696-66e9-4e1d-9419-2597fef50ec7" (UID: "ed5d8696-66e9-4e1d-9419-2597fef50ec7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:21.253158 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.253104 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-workload-certs" pod "ed5d8696-66e9-4e1d-9419-2597fef50ec7" (UID: "ed5d8696-66e9-4e1d-9419-2597fef50ec7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:21.302454 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.300812 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-token\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:21.420221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.405374 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-credential-socket" pod "ed5d8696-66e9-4e1d-9419-2597fef50ec7" (UID: "ed5d8696-66e9-4e1d-9419-2597fef50ec7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:21.501271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.461471 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-data\") pod \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " Oct 29 23:03:21.504398 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.504372 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-envoy\") pod \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " Oct 29 23:03:21.515552 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.515494 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-podinfo\") pod \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " Oct 29 23:03:21.515925 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.515888 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-token\" (UniqueName: \"kubernetes.io/projected/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-token\") pod \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " Oct 29 23:03:21.532885 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.532843 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-workload-socket\") pod \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\" (UID: \"ed5d8696-66e9-4e1d-9419-2597fef50ec7\") " Oct 29 23:03:21.609651 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.609477 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.609651 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.609513 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.609651 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.609529 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.609914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.477507 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-data" pod "ed5d8696-66e9-4e1d-9419-2597fef50ec7" (UID: "ed5d8696-66e9-4e1d-9419-2597fef50ec7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:21.609914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.604465 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-workload-socket" pod "ed5d8696-66e9-4e1d-9419-2597fef50ec7" (UID: "ed5d8696-66e9-4e1d-9419-2597fef50ec7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:21.609914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.609818 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-envoy" pod "ed5d8696-66e9-4e1d-9419-2597fef50ec7" (UID: "ed5d8696-66e9-4e1d-9419-2597fef50ec7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:21.638813 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.625894 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-token" pod "ed5d8696-66e9-4e1d-9419-2597fef50ec7" (UID: "ed5d8696-66e9-4e1d-9419-2597fef50ec7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:21.651500 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.651450 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-podinfo" pod "ed5d8696-66e9-4e1d-9419-2597fef50ec7" (UID: "ed5d8696-66e9-4e1d-9419-2597fef50ec7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:03:21.659718 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.659269 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 23:03:21.678028 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.677972 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5d8696-66e9-4e1d-9419-2597fef50ec7-kube-api-access-r9kbv" pod "ed5d8696-66e9-4e1d-9419-2597fef50ec7" (UID: "ed5d8696-66e9-4e1d-9419-2597fef50ec7"). InnerVolumeSpecName "kube-api-access-r9kbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:21.678839 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.678765 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-envoy\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:21.681531 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.681471 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-podinfo\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:21.693433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.693399 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" event={"ID":"d205536f-2f06-4f8a-9ef1-e640f0904f6c","Type":"ContainerDied","Data":"77bfe2b7a7994b7d8abc25b77a885ce283369b5316c9cd97f1b589cbe1e27711"} Oct 29 23:03:21.693646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.693629 5057 scope.go:122] "RemoveContainer" containerID="565e7676cf8de179628f79102d9f0f0b9fda2fbeac48cfcb067a103aef2dc4ce" Oct 29 23:03:21.693911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.693891 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" Oct 29 23:03:21.695920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.695900 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" Oct 29 23:03:21.699441 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.699413 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" event={"ID":"f2b7f715-3c33-483f-9d7c-75797d057663","Type":"ContainerDied","Data":"36813a7f68a19e92ff2bcd4f3a62970df9bcab57642834fe393d1edc7d526792"} Oct 29 23:03:21.701879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.701833 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" event={"ID":"ed5d8696-66e9-4e1d-9419-2597fef50ec7","Type":"ContainerDied","Data":"c26b7b58392af387b8141ade178425f3f8b439d471935fb953dab455e545b2a7"} Oct 29 23:03:21.702917 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.701991 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj" Oct 29 23:03:21.702917 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.702844 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 23:03:21.708517 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.708367 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk" event={"ID":"356ee5cf-6fec-47b4-90c1-0c1354b74c18","Type":"ContainerStarted","Data":"8b2e7f0210e7c2cf8c0f8dc9bc3a922b9be3ba7d3c6bbf55cdf1d0dfb22079a0"} Oct 29 23:03:21.712263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.711889 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.712263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.711915 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.712263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.711930 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.712263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.711943 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed5d8696-66e9-4e1d-9419-2597fef50ec7-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.712263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.711956 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5d8696-66e9-4e1d-9419-2597fef50ec7-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.712263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.711969 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9kbv\" (UniqueName: \"kubernetes.io/projected/ed5d8696-66e9-4e1d-9419-2597fef50ec7-kube-api-access-r9kbv\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.733283 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.733251 5057 scope.go:122] "RemoveContainer" containerID="f7fe21a8787828e18c2397f8a14611827f3ac93282462d103edcdee4f1926625" Oct 29 23:03:21.748397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.748373 5057 scope.go:122] "RemoveContainer" containerID="5bbef6450b52a2881cecb2cd8432e4ef841f31ac3722466986bc1f46107a55bf" Oct 29 23:03:21.787936 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.787909 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj"] Oct 29 23:03:21.792032 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.792003 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-tk4sj"] Oct 29 23:03:21.813075 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.813043 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istiod-ca-cert\") pod \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " Oct 29 23:03:21.813249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.813083 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-credential-socket\") pod \"e0c58b99-cc75-42a7-8ba1-8734214ca340\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " Oct 29 23:03:21.813249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.813113 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c58b99-cc75-42a7-8ba1-8734214ca340-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0c58b99-cc75-42a7-8ba1-8734214ca340-istiod-ca-cert\") pod \"e0c58b99-cc75-42a7-8ba1-8734214ca340\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " Oct 29 23:03:21.813249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.813202 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b7f715-3c33-483f-9d7c-75797d057663-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b7f715-3c33-483f-9d7c-75797d057663-openshift-service-ca\") pod \"f2b7f715-3c33-483f-9d7c-75797d057663\" (UID: \"f2b7f715-3c33-483f-9d7c-75797d057663\") " Oct 29 23:03:21.813437 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.813279 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-workload-certs\") pod \"e0c58b99-cc75-42a7-8ba1-8734214ca340\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " Oct 29 23:03:21.813437 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.813331 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b7f715-3c33-483f-9d7c-75797d057663-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b7f715-3c33-483f-9d7c-75797d057663-maas-api-tls\") pod \"f2b7f715-3c33-483f-9d7c-75797d057663\" (UID: \"f2b7f715-3c33-483f-9d7c-75797d057663\") " Oct 29 23:03:21.813437 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.813366 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-credential-socket\") pod \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " Oct 29 23:03:21.813595 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.813455 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istiod-ca-cert" pod "d205536f-2f06-4f8a-9ef1-e640f0904f6c" (UID: "d205536f-2f06-4f8a-9ef1-e640f0904f6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:21.813595 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.813472 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-podinfo\") pod \"e0c58b99-cc75-42a7-8ba1-8734214ca340\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " Oct 29 23:03:21.813595 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.813514 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b7f715-3c33-483f-9d7c-75797d057663-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b7f715-3c33-483f-9d7c-75797d057663-metrics-certs\") pod \"f2b7f715-3c33-483f-9d7c-75797d057663\" (UID: \"f2b7f715-3c33-483f-9d7c-75797d057663\") " Oct 29 23:03:21.813595 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.813572 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-podinfo\") pod \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " Oct 29 23:03:21.813797 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.813627 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d205536f-2f06-4f8a-9ef1-e640f0904f6c-kube-api-access-ljvzr\" (UniqueName: \"kubernetes.io/projected/d205536f-2f06-4f8a-9ef1-e640f0904f6c-kube-api-access-ljvzr\") pod \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " Oct 29 23:03:21.813797 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.813698 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-workload-socket\") pod \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " Oct 29 23:03:21.813907 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.813844 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-data\") pod \"e0c58b99-cc75-42a7-8ba1-8734214ca340\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " Oct 29 23:03:21.813907 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.813900 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-envoy\") pod \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " Oct 29 23:03:21.814011 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.813942 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-data\") pod \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " Oct 29 23:03:21.814011 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.813969 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b7f715-3c33-483f-9d7c-75797d057663-openshift-service-ca" pod "f2b7f715-3c33-483f-9d7c-75797d057663" (UID: "f2b7f715-3c33-483f-9d7c-75797d057663"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:21.814011 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.813978 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-workload-socket\") pod \"e0c58b99-cc75-42a7-8ba1-8734214ca340\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " Oct 29 23:03:21.814164 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.814065 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-token\" (UniqueName: \"kubernetes.io/projected/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-token\") pod \"e0c58b99-cc75-42a7-8ba1-8734214ca340\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " Oct 29 23:03:21.814164 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.814134 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-token\" (UniqueName: \"kubernetes.io/projected/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-token\") pod \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " Oct 29 23:03:21.814400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.814179 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-workload-certs\") pod \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\" (UID: \"d205536f-2f06-4f8a-9ef1-e640f0904f6c\") " Oct 29 23:03:21.814400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.814236 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-credential-socket" pod "e0c58b99-cc75-42a7-8ba1-8734214ca340" (UID: "e0c58b99-cc75-42a7-8ba1-8734214ca340"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:21.814400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.814252 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c58b99-cc75-42a7-8ba1-8734214ca340-kube-api-access-mrqjh\" (UniqueName: \"kubernetes.io/projected/e0c58b99-cc75-42a7-8ba1-8734214ca340-kube-api-access-mrqjh\") pod \"e0c58b99-cc75-42a7-8ba1-8734214ca340\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " Oct 29 23:03:21.814575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.814415 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-envoy\") pod \"e0c58b99-cc75-42a7-8ba1-8734214ca340\" (UID: \"e0c58b99-cc75-42a7-8ba1-8734214ca340\") " Oct 29 23:03:21.814575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.814464 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b7f715-3c33-483f-9d7c-75797d057663-kube-api-access-fnxkj\" (UniqueName: \"kubernetes.io/projected/f2b7f715-3c33-483f-9d7c-75797d057663-kube-api-access-fnxkj\") pod \"f2b7f715-3c33-483f-9d7c-75797d057663\" (UID: \"f2b7f715-3c33-483f-9d7c-75797d057663\") " Oct 29 23:03:21.814575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.814532 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c58b99-cc75-42a7-8ba1-8734214ca340-istiod-ca-cert" pod "e0c58b99-cc75-42a7-8ba1-8734214ca340" (UID: "e0c58b99-cc75-42a7-8ba1-8734214ca340"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:21.815383 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.815082 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-data" pod "e0c58b99-cc75-42a7-8ba1-8734214ca340" (UID: "e0c58b99-cc75-42a7-8ba1-8734214ca340"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:21.815649 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.815601 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-credential-socket" pod "d205536f-2f06-4f8a-9ef1-e640f0904f6c" (UID: "d205536f-2f06-4f8a-9ef1-e640f0904f6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:21.815797 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.815773 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-workload-certs" pod "e0c58b99-cc75-42a7-8ba1-8734214ca340" (UID: "e0c58b99-cc75-42a7-8ba1-8734214ca340"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:21.816033 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.816012 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.816101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.816033 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.816101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.816043 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0c58b99-cc75-42a7-8ba1-8734214ca340-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.816101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.816052 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b7f715-3c33-483f-9d7c-75797d057663-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.816101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.816061 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.816101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.816077 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.816101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.816095 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.816448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.816175 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-workload-socket" pod "d205536f-2f06-4f8a-9ef1-e640f0904f6c" (UID: "d205536f-2f06-4f8a-9ef1-e640f0904f6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:21.816448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.816347 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-data" pod "d205536f-2f06-4f8a-9ef1-e640f0904f6c" (UID: "d205536f-2f06-4f8a-9ef1-e640f0904f6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:21.816550 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.816473 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-workload-socket" pod "e0c58b99-cc75-42a7-8ba1-8734214ca340" (UID: "e0c58b99-cc75-42a7-8ba1-8734214ca340"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:21.819554 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.819298 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-workload-certs" pod "d205536f-2f06-4f8a-9ef1-e640f0904f6c" (UID: "d205536f-2f06-4f8a-9ef1-e640f0904f6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:21.820101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.820040 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-podinfo" pod "e0c58b99-cc75-42a7-8ba1-8734214ca340" (UID: "e0c58b99-cc75-42a7-8ba1-8734214ca340"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:03:21.821769 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.821329 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c58b99-cc75-42a7-8ba1-8734214ca340-kube-api-access-mrqjh" pod "e0c58b99-cc75-42a7-8ba1-8734214ca340" (UID: "e0c58b99-cc75-42a7-8ba1-8734214ca340"). InnerVolumeSpecName "kube-api-access-mrqjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:21.821769 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.821644 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-envoy" pod "d205536f-2f06-4f8a-9ef1-e640f0904f6c" (UID: "d205536f-2f06-4f8a-9ef1-e640f0904f6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:21.821769 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.821702 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-podinfo" pod "d205536f-2f06-4f8a-9ef1-e640f0904f6c" (UID: "d205536f-2f06-4f8a-9ef1-e640f0904f6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:03:21.823323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.822084 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d205536f-2f06-4f8a-9ef1-e640f0904f6c-kube-api-access-ljvzr" pod "d205536f-2f06-4f8a-9ef1-e640f0904f6c" (UID: "d205536f-2f06-4f8a-9ef1-e640f0904f6c"). InnerVolumeSpecName "kube-api-access-ljvzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:21.823875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.823689 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-token" pod "d205536f-2f06-4f8a-9ef1-e640f0904f6c" (UID: "d205536f-2f06-4f8a-9ef1-e640f0904f6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:21.823875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.823755 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-token" pod "e0c58b99-cc75-42a7-8ba1-8734214ca340" (UID: "e0c58b99-cc75-42a7-8ba1-8734214ca340"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:21.823875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.823810 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b7f715-3c33-483f-9d7c-75797d057663-kube-api-access-fnxkj" pod "f2b7f715-3c33-483f-9d7c-75797d057663" (UID: "f2b7f715-3c33-483f-9d7c-75797d057663"). InnerVolumeSpecName "kube-api-access-fnxkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:21.823875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.823863 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b7f715-3c33-483f-9d7c-75797d057663-metrics-certs" pod "f2b7f715-3c33-483f-9d7c-75797d057663" (UID: "f2b7f715-3c33-483f-9d7c-75797d057663"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:21.824319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.824289 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b7f715-3c33-483f-9d7c-75797d057663-maas-api-tls" pod "f2b7f715-3c33-483f-9d7c-75797d057663" (UID: "f2b7f715-3c33-483f-9d7c-75797d057663"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:21.827583 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.827552 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-envoy" pod "e0c58b99-cc75-42a7-8ba1-8734214ca340" (UID: "e0c58b99-cc75-42a7-8ba1-8734214ca340"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:21.860373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.858029 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq"] Oct 29 23:03:21.922761 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.922727 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.922761 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.922757 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.922761 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.922765 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.923098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.922773 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.923098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.922781 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.923098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.922790 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.923098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.922797 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d205536f-2f06-4f8a-9ef1-e640f0904f6c-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.923098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.922806 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrqjh\" (UniqueName: \"kubernetes.io/projected/e0c58b99-cc75-42a7-8ba1-8734214ca340-kube-api-access-mrqjh\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.923098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.922814 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.923098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.922824 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnxkj\" (UniqueName: \"kubernetes.io/projected/f2b7f715-3c33-483f-9d7c-75797d057663-kube-api-access-fnxkj\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.923098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.922832 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b7f715-3c33-483f-9d7c-75797d057663-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.923098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.922840 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0c58b99-cc75-42a7-8ba1-8734214ca340-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.923098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.922848 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b7f715-3c33-483f-9d7c-75797d057663-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.923098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.922855 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d205536f-2f06-4f8a-9ef1-e640f0904f6c-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.923098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.922863 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljvzr\" (UniqueName: \"kubernetes.io/projected/d205536f-2f06-4f8a-9ef1-e640f0904f6c-kube-api-access-ljvzr\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.943885 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:21.943851 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:22.095299 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.095215 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7"] Oct 29 23:03:22.109407 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.104130 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7"] Oct 29 23:03:22.364024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.363917 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc"] Oct 29 23:03:22.365543 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.365505 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:03:22.365652 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.365585 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:03:22.365652 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.365632 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:03:22.623385 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.623293 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:03:22.623385 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.623354 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-gjzf7" podUID="d205536f-2f06-4f8a-9ef1-e640f0904f6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:22.721045 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.721014 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx" Oct 29 23:03:22.747130 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.747087 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk" event={"ID":"356ee5cf-6fec-47b4-90c1-0c1354b74c18","Type":"ContainerStarted","Data":"165218c229ef1c8fd64e58950c0b18841dd2b6336a2c97174e0f1850aabde700"} Oct 29 23:03:22.748127 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.748102 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk" Oct 29 23:03:22.750324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.750300 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq" event={"ID":"b49892fa-80a0-4f7b-a75d-fb027e183254","Type":"ContainerStarted","Data":"6521468e891f10a4226c91c6d742b40f9deabd014d28b5620750730d00971815"} Oct 29 23:03:22.750405 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.750348 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq" event={"ID":"b49892fa-80a0-4f7b-a75d-fb027e183254","Type":"ContainerStarted","Data":"921b3e8fa06a2a1c71d7c5438d31f2201d8ed0acbba376a60d4f0a2dfea6842a"} Oct 29 23:03:22.751162 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.751141 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq" Oct 29 23:03:22.753219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.753163 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" event={"ID":"e0c58b99-cc75-42a7-8ba1-8734214ca340","Type":"ContainerDied","Data":"3fd7e1e52fa687b2b1d1192e48d8b3a4a2f85a9203921cc46330541515f4e312"} Oct 29 23:03:22.753346 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.753229 5057 scope.go:122] "RemoveContainer" containerID="2fb7a38252a53a5b475c35550ff5beb9cdbbd5fdb1707f81621892c0d3f0800b" Oct 29 23:03:22.753409 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.753375 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6" Oct 29 23:03:22.768367 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.768333 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" event={"ID":"bbfc0179-11dd-431d-af1a-0793c48ba6fe","Type":"ContainerStarted","Data":"cdc998dc73604bdd420660d1c09d263e9d4eb611a87e24c91785b95080835cbf"} Oct 29 23:03:22.768557 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.768537 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" event={"ID":"bbfc0179-11dd-431d-af1a-0793c48ba6fe","Type":"ContainerStarted","Data":"74bdf25bab4334f36a532dc61d861968c2f75187525a3392a8a814aff46a80df"} Oct 29 23:03:22.793210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.790773 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx"] Oct 29 23:03:22.800275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.799427 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7fa011-79bbf99cd-mvnvx"] Oct 29 23:03:22.872813 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.872755 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" podStartSLOduration=2.872740918 podStartE2EDuration="2.872740918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:22.87253408 +0000 UTC m=+1249.524032612" watchObservedRunningTime="2026-10-29 23:03:22.872740918 +0000 UTC m=+1249.524239489" Oct 29 23:03:22.919448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.919342 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq" podStartSLOduration=3.919326772 podStartE2EDuration="3.919326772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:22.915172163 +0000 UTC m=+1249.566670694" watchObservedRunningTime="2026-10-29 23:03:22.919326772 +0000 UTC m=+1249.570825301" Oct 29 23:03:22.944283 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.944251 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:22.965852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:22.965795 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk" podStartSLOduration=3.965777371 podStartE2EDuration="3.965777371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:22.963393155 +0000 UTC m=+1249.614891683" watchObservedRunningTime="2026-10-29 23:03:22.965777371 +0000 UTC m=+1249.617275902" Oct 29 23:03:23.005962 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:23.005919 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6"] Oct 29 23:03:23.010118 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:23.010094 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-8nrb6"] Oct 29 23:03:23.568206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:23.563526 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d205536f-2f06-4f8a-9ef1-e640f0904f6c" path="/var/lib/kubelet/pods/d205536f-2f06-4f8a-9ef1-e640f0904f6c/volumes" Oct 29 23:03:23.568206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:23.564291 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0c58b99-cc75-42a7-8ba1-8734214ca340" path="/var/lib/kubelet/pods/e0c58b99-cc75-42a7-8ba1-8734214ca340/volumes" Oct 29 23:03:23.568206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:23.564940 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed5d8696-66e9-4e1d-9419-2597fef50ec7" path="/var/lib/kubelet/pods/ed5d8696-66e9-4e1d-9419-2597fef50ec7/volumes" Oct 29 23:03:23.568206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:23.566290 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b7f715-3c33-483f-9d7c-75797d057663" path="/var/lib/kubelet/pods/f2b7f715-3c33-483f-9d7c-75797d057663/volumes" Oct 29 23:03:23.952209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:23.946988 5057 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:03:23.952209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:23.947054 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" podUID="bbfc0179-11dd-431d-af1a-0793c48ba6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 29 23:03:24.944632 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:24.944595 5057 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:03:24.944907 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:24.944880 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" podUID="bbfc0179-11dd-431d-af1a-0793c48ba6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 29 23:03:25.819400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.806422 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl"] Oct 29 23:03:25.819400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.807734 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f2b7f715-3c33-483f-9d7c-75797d057663" containerName="maas-api" Oct 29 23:03:25.819400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.807755 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b7f715-3c33-483f-9d7c-75797d057663" containerName="maas-api" Oct 29 23:03:25.819400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.807780 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d205536f-2f06-4f8a-9ef1-e640f0904f6c" containerName="istio-proxy" Oct 29 23:03:25.819400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.807787 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d205536f-2f06-4f8a-9ef1-e640f0904f6c" containerName="istio-proxy" Oct 29 23:03:25.819400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.807835 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e0c58b99-cc75-42a7-8ba1-8734214ca340" containerName="istio-proxy" Oct 29 23:03:25.819400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.807842 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c58b99-cc75-42a7-8ba1-8734214ca340" containerName="istio-proxy" Oct 29 23:03:25.819400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.807906 5057 cpu_manager.go:414] "containerMap: removing container" podUID="ed5d8696-66e9-4e1d-9419-2597fef50ec7" containerName="istio-proxy" Oct 29 23:03:25.819400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.807915 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed5d8696-66e9-4e1d-9419-2597fef50ec7" containerName="istio-proxy" Oct 29 23:03:25.819400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.808124 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed5d8696-66e9-4e1d-9419-2597fef50ec7" containerName="istio-proxy" Oct 29 23:03:25.819400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.808141 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b7f715-3c33-483f-9d7c-75797d057663" containerName="maas-api" Oct 29 23:03:25.819400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.808153 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d205536f-2f06-4f8a-9ef1-e640f0904f6c" containerName="istio-proxy" Oct 29 23:03:25.819400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.808171 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c58b99-cc75-42a7-8ba1-8734214ca340" containerName="istio-proxy" Oct 29 23:03:25.819400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.809310 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:03:25.819400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.809358 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" podUID="10e45f5c-eae2-4876-ab74-f5949bea38e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:25.819400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.815792 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:25.887300 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.887265 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2"] Oct 29 23:03:25.892892 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.892802 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-44a4e4-openshift-default-dockercfg-qjfbh\"" Oct 29 23:03:25.938523 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.938490 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl"] Oct 29 23:03:25.938883 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.938864 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:25.972225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.960531 5057 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:03:25.974232 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.974175 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" podUID="bbfc0179-11dd-431d-af1a-0793c48ba6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 29 23:03:25.974455 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.973247 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2"] Oct 29 23:03:25.999679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.998414 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-workload-certs\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:25.999679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.998479 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-token\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:25.999679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.998582 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-workload-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:25.999679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.998636 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxhcf\" (UniqueName: \"kubernetes.io/projected/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-kube-api-access-pxhcf\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:25.999679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.998685 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-workload-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:25.999679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.998747 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istiod-ca-cert\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:25.999679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.998781 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-envoy\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:25.999679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.998803 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istiod-ca-cert\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:25.999679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.998826 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-data\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:25.999679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.998864 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-workload-certs\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:25.999679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.998901 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-envoy\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:25.999679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.998936 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-podinfo\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:25.999679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.998959 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-credential-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:25.999679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.998986 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-data\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:25.999679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.999036 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-token\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:25.999679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.999393 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvjnf\" (UniqueName: \"kubernetes.io/projected/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-kube-api-access-xvjnf\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:25.999679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.999462 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-podinfo\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:25.999679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:25.999498 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-credential-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:26.104218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.104119 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-workload-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:26.104407 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.104266 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxhcf\" (UniqueName: \"kubernetes.io/projected/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-kube-api-access-pxhcf\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:26.104407 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.104305 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-workload-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:26.104540 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.104439 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istiod-ca-cert\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:26.104540 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.104464 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-envoy\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:26.104648 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.104572 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istiod-ca-cert\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:26.104648 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.104596 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-data\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:26.104648 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.104623 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-workload-certs\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:26.104856 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.104742 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-envoy\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:26.104856 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.104772 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-podinfo\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:26.104856 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.104786 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-credential-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:26.105050 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.104804 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-data\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:26.105050 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.105002 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-token\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:26.105150 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.105079 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvjnf\" (UniqueName: \"kubernetes.io/projected/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-kube-api-access-xvjnf\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:26.105150 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.105128 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-podinfo\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:26.105286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.105156 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-credential-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:26.105674 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.105651 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-workload-certs\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:26.105756 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.105694 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-token\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:26.111411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.109053 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-workload-certs\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:26.111411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.109470 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-credential-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:26.111411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.109705 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-data\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:26.111411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.105512 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-workload-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:26.111411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.110465 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-credential-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:26.111411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.110660 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-workload-certs\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:26.111411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.110842 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-workload-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:26.111411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.111356 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istiod-ca-cert\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:26.112414 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.111928 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-envoy\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:26.112414 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.112347 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-podinfo\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:26.112585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.112529 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-data\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:26.112677 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.112649 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istiod-ca-cert\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:26.113948 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.113900 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-podinfo\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:26.114095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.114069 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-envoy\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:26.170336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.170294 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxhcf\" (UniqueName: \"kubernetes.io/projected/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-kube-api-access-pxhcf\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:26.172307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.171868 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-token\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:26.172307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.172004 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvjnf\" (UniqueName: \"kubernetes.io/projected/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-kube-api-access-xvjnf\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:26.173609 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.173582 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-token\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:26.258626 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.258579 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:26.304441 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.304408 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:26.608774 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:03:26.608741 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7fa998_f3a7_4ef0_85f2_8662c3fe8bbb.slice/crio-c88e073f155e25e39694077a1532d23709991b69b73ad009a0cebf935b7ed7cf WatchSource:0}: Error finding container c88e073f155e25e39694077a1532d23709991b69b73ad009a0cebf935b7ed7cf: Status 404 returned error can't find the container with id c88e073f155e25e39694077a1532d23709991b69b73ad009a0cebf935b7ed7cf Oct 29 23:03:26.610098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.610074 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl"] Oct 29 23:03:26.612181 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.612146 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:03:26.612259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.612240 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:03:26.612292 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.612273 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:03:26.748972 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.748934 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2"] Oct 29 23:03:26.781674 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.781264 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:03:26.781674 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.781373 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:03:26.781674 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.781425 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:03:26.945871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.945788 5057 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:03:26.946578 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:26.946534 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" podUID="bbfc0179-11dd-431d-af1a-0793c48ba6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 29 23:03:27.020617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:27.020578 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" event={"ID":"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb","Type":"ContainerStarted","Data":"187c6bdb9dc820e5d86851bd496c91dccb4b8594c6a46c68ff070ea5db833a15"} Oct 29 23:03:27.020870 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:27.020832 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" event={"ID":"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb","Type":"ContainerStarted","Data":"c88e073f155e25e39694077a1532d23709991b69b73ad009a0cebf935b7ed7cf"} Oct 29 23:03:27.036339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:27.036232 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" event={"ID":"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992","Type":"ContainerStarted","Data":"888bb25ab3c5c9bbd08d6d6a6c94af0c3c83519e8700cda85aff20679b9b85a2"} Oct 29 23:03:27.260493 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:27.213096 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" podStartSLOduration=2.213075032 podStartE2EDuration="2.213075032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:27.159140759 +0000 UTC m=+1253.810639290" watchObservedRunningTime="2026-10-29 23:03:27.213075032 +0000 UTC m=+1253.864573564" Oct 29 23:03:27.262425 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:27.262320 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:27.303236 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:27.302501 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 23:03:27.303236 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:27.302576 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" podUID="0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 23:03:28.004829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:27.947831 5057 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:03:28.004829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:27.947891 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" podUID="bbfc0179-11dd-431d-af1a-0793c48ba6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 29 23:03:28.102209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:28.092262 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" event={"ID":"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992","Type":"ContainerStarted","Data":"2627aec871c546110e0c470c9df283c45a4b93ac99daca3be28c1ec29948fc0b"} Oct 29 23:03:28.176444 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:28.176371 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" podStartSLOduration=3.176347874 podStartE2EDuration="3.176347874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:28.173803435 +0000 UTC m=+1254.825301965" watchObservedRunningTime="2026-10-29 23:03:28.176347874 +0000 UTC m=+1254.827846405" Oct 29 23:03:28.259726 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:28.259638 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 23:03:28.260075 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:28.260042 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" podUID="0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 23:03:28.309697 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:28.309666 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:28.325356 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:28.321031 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:03:28.325356 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:28.321104 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" podUID="2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 29 23:03:28.991180 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:28.989328 5057 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:03:28.991180 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:28.989390 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" podUID="bbfc0179-11dd-431d-af1a-0793c48ba6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 29 23:03:29.262263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:29.261727 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 23:03:29.262263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:29.261786 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" podUID="0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 23:03:29.307212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:29.307073 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:03:29.307212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:29.307130 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" podUID="2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 29 23:03:29.969211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:29.968870 5057 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:03:29.969211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:29.968943 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" podUID="bbfc0179-11dd-431d-af1a-0793c48ba6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 29 23:03:30.345257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:30.344517 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:03:30.345257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:30.344579 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" podUID="2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 29 23:03:30.881215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:30.802576 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 23:03:30.881542 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:30.881498 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" podUID="0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 23:03:31.114271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.105401 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 23:03:31.312367 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.312335 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx"] Oct 29 23:03:31.317335 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.317307 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:03:31.317591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.317565 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" podUID="2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 29 23:03:31.319407 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.319377 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 23:03:31.319535 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.319422 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" podUID="0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 23:03:31.321019 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.320997 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:31.322661 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.321132 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.322926 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.322885 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:31.329274 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.329252 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:03:31.342820 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.341483 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7af1a5-openshift-default-dockercfg-5hnvl\"" Oct 29 23:03:31.374432 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.374392 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx"] Oct 29 23:03:31.413024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.412996 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-istio-envoy\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.413239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.413223 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f525a668-448c-44ed-8fac-21db37da4c29-istio-token\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.413333 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.413323 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f525a668-448c-44ed-8fac-21db37da4c29-istio-podinfo\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.413426 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.413417 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f525a668-448c-44ed-8fac-21db37da4c29-istiod-ca-cert\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.413500 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.413491 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-workload-socket\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.413561 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.413551 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzzrk\" (UniqueName: \"kubernetes.io/projected/f525a668-448c-44ed-8fac-21db37da4c29-kube-api-access-lzzrk\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.413696 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.413686 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-credential-socket\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.414275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.414259 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-istio-data\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.414591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.414572 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-workload-certs\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.517459 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.517421 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-istio-envoy\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.517636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.517468 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f525a668-448c-44ed-8fac-21db37da4c29-istio-token\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.517636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.517508 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f525a668-448c-44ed-8fac-21db37da4c29-istio-podinfo\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.518234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.518209 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f525a668-448c-44ed-8fac-21db37da4c29-istiod-ca-cert\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.518349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.518249 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-workload-socket\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.518349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.518267 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzzrk\" (UniqueName: \"kubernetes.io/projected/f525a668-448c-44ed-8fac-21db37da4c29-kube-api-access-lzzrk\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.518460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.518391 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-credential-socket\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.518523 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.518489 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-istio-data\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.518746 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.518724 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-workload-certs\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.519843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.519814 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f525a668-448c-44ed-8fac-21db37da4c29-istiod-ca-cert\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.519843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.519829 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-workload-socket\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.520138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.520114 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-credential-socket\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.520402 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.520378 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-istio-data\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.520568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.520543 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-workload-certs\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.530214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.521594 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f525a668-448c-44ed-8fac-21db37da4c29-istio-podinfo\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.553223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.552474 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-istio-envoy\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.590976 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.590826 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzzrk\" (UniqueName: \"kubernetes.io/projected/f525a668-448c-44ed-8fac-21db37da4c29-kube-api-access-lzzrk\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.611630 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.611579 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f525a668-448c-44ed-8fac-21db37da4c29-istio-token\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:31.735732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:31.735696 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:32.309140 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:32.264145 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 23:03:32.309140 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:32.264239 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" podUID="0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 23:03:32.411220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:32.385826 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:03:32.507717 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:32.507534 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" podUID="2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 29 23:03:32.809269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:32.803903 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx"] Oct 29 23:03:33.086220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:33.077712 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:03:33.086683 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:33.086643 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:03:33.086904 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:33.086881 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:03:33.310862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:33.310828 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:03:33.311128 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:33.311100 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" podUID="2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 29 23:03:33.315208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:33.311977 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 23:03:33.315630 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:33.315602 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" podUID="0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 23:03:33.802250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:33.802220 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" event={"ID":"f525a668-448c-44ed-8fac-21db37da4c29","Type":"ContainerStarted","Data":"e5c04b3689e1fd7921138ea678c9d04e47da8e494b41f7cea6e8b884f8d118a6"} Oct 29 23:03:34.284211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:34.272826 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 23:03:34.284211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:34.272900 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" podUID="0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 23:03:34.380213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:34.372876 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:03:34.380213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:34.372931 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" podUID="2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 29 23:03:34.581438 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:03:34.580410 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:03:34.633217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:34.613294 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv"] Oct 29 23:03:34.702211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:34.685414 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:34.760215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:34.734486 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-credential-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:34.767240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:34.760601 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b0c72c-openshift-default-dockercfg-twhkp\"" Oct 29 23:03:34.819329 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:34.819286 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d728ed56-8f9a-4396-8386-7ce602570421-istio-podinfo\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.230026 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:34.895361 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk" Oct 29 23:03:35.230436 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.230406 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq" Oct 29 23:03:35.230622 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.230607 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv"] Oct 29 23:03:35.279591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.270371 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-workload-certs\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.279591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.270633 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-workload-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.279591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.270670 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-istio-data\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.279591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.270721 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-istio-envoy\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.279591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.270849 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d728ed56-8f9a-4396-8386-7ce602570421-istio-token\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.279591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.270893 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d728ed56-8f9a-4396-8386-7ce602570421-istiod-ca-cert\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.279591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.270926 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p6jg\" (UniqueName: \"kubernetes.io/projected/d728ed56-8f9a-4396-8386-7ce602570421-kube-api-access-7p6jg\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.320768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.320731 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 23:03:35.321008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.320977 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" podUID="0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 23:03:35.344421 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.344380 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" event={"ID":"f525a668-448c-44ed-8fac-21db37da4c29","Type":"ContainerStarted","Data":"3a15c28171201f3e2e413b776c629c6caeccc5bd03f816270cf1b4fb9e0f80e0"} Oct 29 23:03:35.376822 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.376786 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-istio-envoy\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.377454 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.377424 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d728ed56-8f9a-4396-8386-7ce602570421-istio-token\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.381345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.381322 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d728ed56-8f9a-4396-8386-7ce602570421-istiod-ca-cert\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.381581 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.381561 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p6jg\" (UniqueName: \"kubernetes.io/projected/d728ed56-8f9a-4396-8386-7ce602570421-kube-api-access-7p6jg\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.381797 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.381775 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-credential-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.381972 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.381956 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d728ed56-8f9a-4396-8386-7ce602570421-istio-podinfo\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.382233 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.382214 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-workload-certs\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.383062 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.383028 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-workload-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.383214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.383181 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-istio-data\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.383805 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.383785 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-istio-data\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.383930 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.378305 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:03:35.384060 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.384037 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" podUID="2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 29 23:03:35.385221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.385146 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d728ed56-8f9a-4396-8386-7ce602570421-istiod-ca-cert\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.385796 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.385775 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-credential-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.394927 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.394900 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d728ed56-8f9a-4396-8386-7ce602570421-istio-podinfo\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.395751 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.395727 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-istio-envoy\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.396327 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.395951 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-workload-certs\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.396884 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.396809 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-workload-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.411562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.411524 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d728ed56-8f9a-4396-8386-7ce602570421-istio-token\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.417648 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.416911 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p6jg\" (UniqueName: \"kubernetes.io/projected/d728ed56-8f9a-4396-8386-7ce602570421-kube-api-access-7p6jg\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.494626 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.494514 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" podStartSLOduration=5.494494751 podStartE2EDuration="5.494494751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:35.406695637 +0000 UTC m=+1262.058194167" watchObservedRunningTime="2026-10-29 23:03:35.494494751 +0000 UTC m=+1262.145993281" Oct 29 23:03:35.622217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.619574 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:35.780108 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.767131 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:35.814008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:35.813664 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:36.261443 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:36.259653 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 23:03:36.261443 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:36.259713 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" podUID="0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 23:03:36.305175 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:36.305104 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:03:36.305175 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:36.305165 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" podUID="2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 29 23:03:36.406607 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:36.404915 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:36.406607 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:36.405837 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:03:36.964053 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:36.957318 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:03:36.964053 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:36.957409 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:03:36.964053 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:36.957453 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:03:36.985698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:36.985380 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv"] Oct 29 23:03:37.266532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:37.266451 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:37.267018 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:37.266945 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:37.267806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:37.267786 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:03:37.461991 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:37.460239 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" event={"ID":"d728ed56-8f9a-4396-8386-7ce602570421","Type":"ContainerStarted","Data":"18425919557b62eb641feaa3a65c07a14d9d83af576c81ff516db8896d16151d"} Oct 29 23:03:37.461991 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:37.460326 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" event={"ID":"d728ed56-8f9a-4396-8386-7ce602570421","Type":"ContainerStarted","Data":"704b6730b81486f41ae80b330e25e375252dd8c86925c2ca7ccc8360cb4940fe"} Oct 29 23:03:37.461991 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:37.461180 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:37.462692 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:37.462509 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:37.465958 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:37.465937 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:03:37.520151 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:37.520029 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" podStartSLOduration=3.520006649 podStartE2EDuration="3.520006649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:37.512996199 +0000 UTC m=+1264.164494731" watchObservedRunningTime="2026-10-29 23:03:37.520006649 +0000 UTC m=+1264.171505181" Oct 29 23:03:37.633336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:37.630302 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:38.638678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:38.638638 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:39.546492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:39.546462 5057 generic.go:356] "Generic (PLEG): container finished" podID="10e45f5c-eae2-4876-ab74-f5949bea38e1" containerID="4f7b40747316cbceabd34ca9c5d18e21a65a2476fc2bac9f61cf0f662e15603f" exitCode=0 Oct 29 23:03:39.546717 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:39.546542 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" event={"ID":"10e45f5c-eae2-4876-ab74-f5949bea38e1","Type":"ContainerDied","Data":"4f7b40747316cbceabd34ca9c5d18e21a65a2476fc2bac9f61cf0f662e15603f"} Oct 29 23:03:39.547139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:39.547112 5057 scope.go:122] "RemoveContainer" containerID="4f7b40747316cbceabd34ca9c5d18e21a65a2476fc2bac9f61cf0f662e15603f" Oct 29 23:03:39.547317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:39.547219 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:39.548322 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:39.548278 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:03:39.549501 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:39.549467 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:03:39.549597 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:39.549556 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:03:39.549639 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:39.549627 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:03:39.763207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:39.761771 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:03:40.629229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:40.594240 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" event={"ID":"10e45f5c-eae2-4876-ab74-f5949bea38e1","Type":"ContainerStarted","Data":"aed255caba5d36dca54e550ff1ef5a858606089be8558109a895a58ff3e290d9"} Oct 29 23:03:40.655144 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:40.655108 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv"] Oct 29 23:03:40.703430 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:40.700093 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7"] Oct 29 23:03:41.179318 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:41.179286 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:03:41.180222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:41.180165 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 29 23:03:41.180414 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:41.180392 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" podUID="10e45f5c-eae2-4876-ab74-f5949bea38e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 29 23:03:42.244441 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:42.244399 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 29 23:03:42.316370 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:42.296299 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" podUID="10e45f5c-eae2-4876-ab74-f5949bea38e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 29 23:03:43.182875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:43.178310 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 29 23:03:43.182875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:43.178382 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" podUID="10e45f5c-eae2-4876-ab74-f5949bea38e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 29 23:03:44.181209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:44.180818 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 29 23:03:44.181209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:44.180875 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" podUID="10e45f5c-eae2-4876-ab74-f5949bea38e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 29 23:03:45.178824 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:45.178785 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 29 23:03:45.179022 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:45.178839 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" podUID="10e45f5c-eae2-4876-ab74-f5949bea38e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 29 23:03:45.656091 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:45.655998 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv" podUID="97751900-4954-4a2a-bf49-99b702e4a7c2" containerName="payload-processing" containerID="cri-o://bf6440b779b3ba16012554fd4220d8bee29bbf91b41a36aea6798366c974f74e" gracePeriod=25 Oct 29 23:03:45.700953 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:45.700910 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7" podUID="299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0" containerName="payload-pre-processing" containerID="cri-o://580ddcff2119fd4c6d6110f20b93a2298260e37832bd5bb6f86373811a77864c" gracePeriod=25 Oct 29 23:03:46.181047 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.180837 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 29 23:03:46.181047 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.180905 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" podUID="10e45f5c-eae2-4876-ab74-f5949bea38e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 29 23:03:46.197918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.197576 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7" Oct 29 23:03:46.292379 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.292341 5057 generic.go:356] "Generic (PLEG): container finished" podID="299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0" containerID="580ddcff2119fd4c6d6110f20b93a2298260e37832bd5bb6f86373811a77864c" exitCode=0 Oct 29 23:03:46.293777 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.292526 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7" event={"ID":"299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0","Type":"ContainerDied","Data":"580ddcff2119fd4c6d6110f20b93a2298260e37832bd5bb6f86373811a77864c"} Oct 29 23:03:46.294052 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.292638 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7" Oct 29 23:03:46.295244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.294524 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7" event={"ID":"299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0","Type":"ContainerDied","Data":"ca668f412c9696425120496cd09618febd2bc57e3489047b2703f8a4d8a88ecc"} Oct 29 23:03:46.295353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.295107 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv" Oct 29 23:03:46.296300 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.296259 5057 scope.go:122] "RemoveContainer" containerID="580ddcff2119fd4c6d6110f20b93a2298260e37832bd5bb6f86373811a77864c" Oct 29 23:03:46.298143 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.298027 5057 generic.go:356] "Generic (PLEG): container finished" podID="97751900-4954-4a2a-bf49-99b702e4a7c2" containerID="bf6440b779b3ba16012554fd4220d8bee29bbf91b41a36aea6798366c974f74e" exitCode=0 Oct 29 23:03:46.298311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.298288 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv" event={"ID":"97751900-4954-4a2a-bf49-99b702e4a7c2","Type":"ContainerDied","Data":"bf6440b779b3ba16012554fd4220d8bee29bbf91b41a36aea6798366c974f74e"} Oct 29 23:03:46.309851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.309824 5057 scope.go:122] "RemoveContainer" containerID="580ddcff2119fd4c6d6110f20b93a2298260e37832bd5bb6f86373811a77864c" Oct 29 23:03:46.310226 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:03:46.310178 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"580ddcff2119fd4c6d6110f20b93a2298260e37832bd5bb6f86373811a77864c\": container with ID starting with 580ddcff2119fd4c6d6110f20b93a2298260e37832bd5bb6f86373811a77864c not found: ID does not exist" containerID="580ddcff2119fd4c6d6110f20b93a2298260e37832bd5bb6f86373811a77864c" Oct 29 23:03:46.310350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.310331 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"580ddcff2119fd4c6d6110f20b93a2298260e37832bd5bb6f86373811a77864c"} err="failed to get container status \"580ddcff2119fd4c6d6110f20b93a2298260e37832bd5bb6f86373811a77864c\": rpc error: code = NotFound desc = could not find container \"580ddcff2119fd4c6d6110f20b93a2298260e37832bd5bb6f86373811a77864c\": container with ID starting with 580ddcff2119fd4c6d6110f20b93a2298260e37832bd5bb6f86373811a77864c not found: ID does not exist" Oct 29 23:03:46.310420 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.310412 5057 scope.go:122] "RemoveContainer" containerID="bf6440b779b3ba16012554fd4220d8bee29bbf91b41a36aea6798366c974f74e" Oct 29 23:03:46.385785 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.385756 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0-kube-api-access-lhmbg\" (UniqueName: \"kubernetes.io/projected/299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0-kube-api-access-lhmbg\") pod \"299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0\" (UID: \"299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0\") " Oct 29 23:03:46.385949 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.385927 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0-plugins-config-volume\") pod \"299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0\" (UID: \"299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0\") " Oct 29 23:03:46.386540 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.386508 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0-plugins-config-volume" pod "299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0" (UID: "299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:46.386711 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.386679 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:46.388640 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.388617 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0-kube-api-access-lhmbg" pod "299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0" (UID: "299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0"). InnerVolumeSpecName "kube-api-access-lhmbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:46.488308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.488224 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97751900-4954-4a2a-bf49-99b702e4a7c2-kube-api-access-8l5m9\" (UniqueName: \"kubernetes.io/projected/97751900-4954-4a2a-bf49-99b702e4a7c2-kube-api-access-8l5m9\") pod \"97751900-4954-4a2a-bf49-99b702e4a7c2\" (UID: \"97751900-4954-4a2a-bf49-99b702e4a7c2\") " Oct 29 23:03:46.488475 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.488460 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97751900-4954-4a2a-bf49-99b702e4a7c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97751900-4954-4a2a-bf49-99b702e4a7c2-plugins-config-volume\") pod \"97751900-4954-4a2a-bf49-99b702e4a7c2\" (UID: \"97751900-4954-4a2a-bf49-99b702e4a7c2\") " Oct 29 23:03:46.488932 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.488893 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97751900-4954-4a2a-bf49-99b702e4a7c2-plugins-config-volume" pod "97751900-4954-4a2a-bf49-99b702e4a7c2" (UID: "97751900-4954-4a2a-bf49-99b702e4a7c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:46.489258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.489237 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97751900-4954-4a2a-bf49-99b702e4a7c2-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:46.489357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.489262 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhmbg\" (UniqueName: \"kubernetes.io/projected/299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0-kube-api-access-lhmbg\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:46.490546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.490523 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97751900-4954-4a2a-bf49-99b702e4a7c2-kube-api-access-8l5m9" pod "97751900-4954-4a2a-bf49-99b702e4a7c2" (UID: "97751900-4954-4a2a-bf49-99b702e4a7c2"). InnerVolumeSpecName "kube-api-access-8l5m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:46.591911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.591878 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l5m9\" (UniqueName: \"kubernetes.io/projected/97751900-4954-4a2a-bf49-99b702e4a7c2-kube-api-access-8l5m9\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:46.658086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.658053 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7"] Oct 29 23:03:46.672315 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:46.672275 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec4f6d-766666575f-86xx7"] Oct 29 23:03:47.178284 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:47.178249 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 29 23:03:47.178466 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:47.178311 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" podUID="10e45f5c-eae2-4876-ab74-f5949bea38e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 29 23:03:47.475260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:47.465413 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv" event={"ID":"97751900-4954-4a2a-bf49-99b702e4a7c2","Type":"ContainerDied","Data":"bd1d32b144d1bbb3cfaf404126434bfbce1ed879f9f4001b3c60a813fdddf377"} Oct 29 23:03:47.475260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:47.465566 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv" Oct 29 23:03:47.703643 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:47.688519 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0" path="/var/lib/kubelet/pods/299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0/volumes" Oct 29 23:03:47.757040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:47.756955 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv"] Oct 29 23:03:47.770753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:47.770719 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec4f6d-6995c59f5d-2gfvv"] Oct 29 23:03:48.229481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:48.229449 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:03:48.230439 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:48.230409 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:03:48.232014 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:48.231991 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:03:48.529479 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:48.529389 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr"] Oct 29 23:03:48.529688 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:48.529661 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" podUID="9f124749-58f1-491c-b3e8-f1a0d0adb8dc" containerName="istio-proxy" containerID="cri-o://6fb006486828d7a9aeb9263149c6823f408d63081df34bade53ac22b18535453" gracePeriod=30 Oct 29 23:03:49.487542 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:49.487510 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97751900-4954-4a2a-bf49-99b702e4a7c2" path="/var/lib/kubelet/pods/97751900-4954-4a2a-bf49-99b702e4a7c2/volumes" Oct 29 23:03:53.852424 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.852388 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 23:03:53.867962 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.867926 5057 generic.go:356] "Generic (PLEG): container finished" podID="9f124749-58f1-491c-b3e8-f1a0d0adb8dc" containerID="6fb006486828d7a9aeb9263149c6823f408d63081df34bade53ac22b18535453" exitCode=0 Oct 29 23:03:53.868128 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.868082 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" event={"ID":"9f124749-58f1-491c-b3e8-f1a0d0adb8dc","Type":"ContainerDied","Data":"6fb006486828d7a9aeb9263149c6823f408d63081df34bade53ac22b18535453"} Oct 29 23:03:53.868223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.868133 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" event={"ID":"9f124749-58f1-491c-b3e8-f1a0d0adb8dc","Type":"ContainerDied","Data":"20fdbea70e3e36f36f172b53a2fe3221bbbb6179a7ef7fb6bd0d6a23d42867e1"} Oct 29 23:03:53.868223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.868163 5057 scope.go:122] "RemoveContainer" containerID="6fb006486828d7a9aeb9263149c6823f408d63081df34bade53ac22b18535453" Oct 29 23:03:53.868387 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.868370 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr" Oct 29 23:03:53.880044 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.880024 5057 scope.go:122] "RemoveContainer" containerID="82182bc5029c288b02f5fbeeb6f83d83d5efc4d8e3568e39950f8c699877af13" Oct 29 23:03:53.890719 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.890693 5057 scope.go:122] "RemoveContainer" containerID="6fb006486828d7a9aeb9263149c6823f408d63081df34bade53ac22b18535453" Oct 29 23:03:53.891000 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:03:53.890980 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb006486828d7a9aeb9263149c6823f408d63081df34bade53ac22b18535453\": container with ID starting with 6fb006486828d7a9aeb9263149c6823f408d63081df34bade53ac22b18535453 not found: ID does not exist" containerID="6fb006486828d7a9aeb9263149c6823f408d63081df34bade53ac22b18535453" Oct 29 23:03:53.891048 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.891009 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb006486828d7a9aeb9263149c6823f408d63081df34bade53ac22b18535453"} err="failed to get container status \"6fb006486828d7a9aeb9263149c6823f408d63081df34bade53ac22b18535453\": rpc error: code = NotFound desc = could not find container \"6fb006486828d7a9aeb9263149c6823f408d63081df34bade53ac22b18535453\": container with ID starting with 6fb006486828d7a9aeb9263149c6823f408d63081df34bade53ac22b18535453 not found: ID does not exist" Oct 29 23:03:53.891048 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.891026 5057 scope.go:122] "RemoveContainer" containerID="82182bc5029c288b02f5fbeeb6f83d83d5efc4d8e3568e39950f8c699877af13" Oct 29 23:03:53.891342 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:03:53.891314 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82182bc5029c288b02f5fbeeb6f83d83d5efc4d8e3568e39950f8c699877af13\": container with ID starting with 82182bc5029c288b02f5fbeeb6f83d83d5efc4d8e3568e39950f8c699877af13 not found: ID does not exist" containerID="82182bc5029c288b02f5fbeeb6f83d83d5efc4d8e3568e39950f8c699877af13" Oct 29 23:03:53.891417 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.891350 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82182bc5029c288b02f5fbeeb6f83d83d5efc4d8e3568e39950f8c699877af13"} err="failed to get container status \"82182bc5029c288b02f5fbeeb6f83d83d5efc4d8e3568e39950f8c699877af13\": rpc error: code = NotFound desc = could not find container \"82182bc5029c288b02f5fbeeb6f83d83d5efc4d8e3568e39950f8c699877af13\": container with ID starting with 82182bc5029c288b02f5fbeeb6f83d83d5efc4d8e3568e39950f8c699877af13 not found: ID does not exist" Oct 29 23:03:53.965640 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.965609 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-workload-socket\") pod \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " Oct 29 23:03:53.965835 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.965717 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-credential-socket\") pod \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " Oct 29 23:03:53.965835 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.965819 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-podinfo\") pod \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " Oct 29 23:03:53.965967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.965898 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-token\" (UniqueName: \"kubernetes.io/projected/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-token\") pod \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " Oct 29 23:03:53.965967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.965906 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-workload-socket" pod "9f124749-58f1-491c-b3e8-f1a0d0adb8dc" (UID: "9f124749-58f1-491c-b3e8-f1a0d0adb8dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:53.966062 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.965977 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-credential-socket" pod "9f124749-58f1-491c-b3e8-f1a0d0adb8dc" (UID: "9f124749-58f1-491c-b3e8-f1a0d0adb8dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:53.966062 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.966026 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-envoy\") pod \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " Oct 29 23:03:53.966161 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.966079 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-kube-api-access-kq6hp\" (UniqueName: \"kubernetes.io/projected/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-kube-api-access-kq6hp\") pod \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " Oct 29 23:03:53.966280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.966182 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istiod-ca-cert\") pod \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " Oct 29 23:03:53.966353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.966309 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-workload-certs\") pod \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " Oct 29 23:03:53.966353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.966345 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-data\") pod \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\" (UID: \"9f124749-58f1-491c-b3e8-f1a0d0adb8dc\") " Oct 29 23:03:53.966914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.966878 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-workload-certs" pod "9f124749-58f1-491c-b3e8-f1a0d0adb8dc" (UID: "9f124749-58f1-491c-b3e8-f1a0d0adb8dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:53.967116 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.967095 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-data" pod "9f124749-58f1-491c-b3e8-f1a0d0adb8dc" (UID: "9f124749-58f1-491c-b3e8-f1a0d0adb8dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:53.967242 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.967178 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:53.967327 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.967246 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:53.967327 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.967261 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:53.967452 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.967426 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istiod-ca-cert" pod "9f124749-58f1-491c-b3e8-f1a0d0adb8dc" (UID: "9f124749-58f1-491c-b3e8-f1a0d0adb8dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:53.968763 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.968736 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-token" pod "9f124749-58f1-491c-b3e8-f1a0d0adb8dc" (UID: "9f124749-58f1-491c-b3e8-f1a0d0adb8dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:53.968914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.968868 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-envoy" pod "9f124749-58f1-491c-b3e8-f1a0d0adb8dc" (UID: "9f124749-58f1-491c-b3e8-f1a0d0adb8dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:53.969948 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.969923 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-podinfo" pod "9f124749-58f1-491c-b3e8-f1a0d0adb8dc" (UID: "9f124749-58f1-491c-b3e8-f1a0d0adb8dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:03:53.970376 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:53.970350 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-kube-api-access-kq6hp" pod "9f124749-58f1-491c-b3e8-f1a0d0adb8dc" (UID: "9f124749-58f1-491c-b3e8-f1a0d0adb8dc"). InnerVolumeSpecName "kube-api-access-kq6hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:54.069637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:54.069495 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:54.069637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:54.069532 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:54.069637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:54.069546 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:54.069637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:54.069558 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq6hp\" (UniqueName: \"kubernetes.io/projected/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-kube-api-access-kq6hp\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:54.069637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:54.069566 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:54.069637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:54.069576 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f124749-58f1-491c-b3e8-f1a0d0adb8dc-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:03:54.234900 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:54.234860 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr"] Oct 29 23:03:54.246061 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:54.246034 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-df82ee-openshift-default-55977dc9f7-fmpgr"] Oct 29 23:03:54.730221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:54.730170 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh"] Oct 29 23:03:54.730709 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:54.730674 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" podUID="33c7a18b-c654-4401-88da-bd2875dd61c9" containerName="istio-proxy" containerID="cri-o://05f8ed87ac01b9b4f37447b05e30fe91682bbb60886b5c6da81cb2513265c38f" gracePeriod=30 Oct 29 23:03:54.748061 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:54.748030 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8"] Oct 29 23:03:54.748305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:54.748280 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" podUID="352d7948-b3c7-4698-bf83-f7246a134611" containerName="istio-proxy" containerID="cri-o://0e04eea48e01c1ba168effbeb8e3e8923e7e177c6a2d9480e1f2fbd409c6871a" gracePeriod=30 Oct 29 23:03:55.485659 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:55.485630 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f124749-58f1-491c-b3e8-f1a0d0adb8dc" path="/var/lib/kubelet/pods/9f124749-58f1-491c-b3e8-f1a0d0adb8dc/volumes" Oct 29 23:03:56.804229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:56.803596 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:03:56.804229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:03:56.803681 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" podUID="33c7a18b-c654-4401-88da-bd2875dd61c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:04:00.211874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.211825 5057 generic.go:356] "Generic (PLEG): container finished" podID="352d7948-b3c7-4698-bf83-f7246a134611" containerID="0e04eea48e01c1ba168effbeb8e3e8923e7e177c6a2d9480e1f2fbd409c6871a" exitCode=0 Oct 29 23:04:00.212480 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.212040 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" event={"ID":"352d7948-b3c7-4698-bf83-f7246a134611","Type":"ContainerDied","Data":"0e04eea48e01c1ba168effbeb8e3e8923e7e177c6a2d9480e1f2fbd409c6871a"} Oct 29 23:04:00.214143 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.214115 5057 generic.go:356] "Generic (PLEG): container finished" podID="33c7a18b-c654-4401-88da-bd2875dd61c9" containerID="05f8ed87ac01b9b4f37447b05e30fe91682bbb60886b5c6da81cb2513265c38f" exitCode=0 Oct 29 23:04:00.214312 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.214292 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" event={"ID":"33c7a18b-c654-4401-88da-bd2875dd61c9","Type":"ContainerDied","Data":"05f8ed87ac01b9b4f37447b05e30fe91682bbb60886b5c6da81cb2513265c38f"} Oct 29 23:04:00.298706 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.298668 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 23:04:00.349687 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.349649 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 23:04:00.427212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.409403 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33c7a18b-c654-4401-88da-bd2875dd61c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33c7a18b-c654-4401-88da-bd2875dd61c9-istiod-ca-cert\") pod \"33c7a18b-c654-4401-88da-bd2875dd61c9\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " Oct 29 23:04:00.427212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.409571 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-workload-socket\") pod \"33c7a18b-c654-4401-88da-bd2875dd61c9\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " Oct 29 23:04:00.427212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.410353 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33c7a18b-c654-4401-88da-bd2875dd61c9-istiod-ca-cert" pod "33c7a18b-c654-4401-88da-bd2875dd61c9" (UID: "33c7a18b-c654-4401-88da-bd2875dd61c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:00.427212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.410490 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-credential-socket\") pod \"33c7a18b-c654-4401-88da-bd2875dd61c9\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " Oct 29 23:04:00.427212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.417521 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-credential-socket" pod "33c7a18b-c654-4401-88da-bd2875dd61c9" (UID: "33c7a18b-c654-4401-88da-bd2875dd61c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:00.428699 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.428654 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-workload-socket" pod "33c7a18b-c654-4401-88da-bd2875dd61c9" (UID: "33c7a18b-c654-4401-88da-bd2875dd61c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:00.429061 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.429041 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33c7a18b-c654-4401-88da-bd2875dd61c9-kube-api-access-ng9wq\" (UniqueName: \"kubernetes.io/projected/33c7a18b-c654-4401-88da-bd2875dd61c9-kube-api-access-ng9wq\") pod \"33c7a18b-c654-4401-88da-bd2875dd61c9\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " Oct 29 23:04:00.435252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.435221 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-podinfo\") pod \"33c7a18b-c654-4401-88da-bd2875dd61c9\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " Oct 29 23:04:00.436806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.436784 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-token\" (UniqueName: \"kubernetes.io/projected/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-token\") pod \"33c7a18b-c654-4401-88da-bd2875dd61c9\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " Oct 29 23:04:00.442266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.437001 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-envoy\") pod \"33c7a18b-c654-4401-88da-bd2875dd61c9\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " Oct 29 23:04:00.442571 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.442551 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-workload-certs\") pod \"33c7a18b-c654-4401-88da-bd2875dd61c9\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " Oct 29 23:04:00.442985 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.442967 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-workload-certs\") pod \"352d7948-b3c7-4698-bf83-f7246a134611\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " Oct 29 23:04:00.446841 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.446815 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-data\") pod \"33c7a18b-c654-4401-88da-bd2875dd61c9\" (UID: \"33c7a18b-c654-4401-88da-bd2875dd61c9\") " Oct 29 23:04:00.446995 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.446981 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/352d7948-b3c7-4698-bf83-f7246a134611-kube-api-access-2ngsp\" (UniqueName: \"kubernetes.io/projected/352d7948-b3c7-4698-bf83-f7246a134611-kube-api-access-2ngsp\") pod \"352d7948-b3c7-4698-bf83-f7246a134611\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " Oct 29 23:04:00.448227 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.448209 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33c7a18b-c654-4401-88da-bd2875dd61c9-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:00.448345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.448335 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:00.448446 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.448436 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:00.448945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.442878 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-workload-certs" pod "33c7a18b-c654-4401-88da-bd2875dd61c9" (UID: "33c7a18b-c654-4401-88da-bd2875dd61c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:00.461438 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.461361 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-data" pod "33c7a18b-c654-4401-88da-bd2875dd61c9" (UID: "33c7a18b-c654-4401-88da-bd2875dd61c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:00.461741 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.461718 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-workload-certs" pod "352d7948-b3c7-4698-bf83-f7246a134611" (UID: "352d7948-b3c7-4698-bf83-f7246a134611"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:00.553303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.544579 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-envoy" pod "33c7a18b-c654-4401-88da-bd2875dd61c9" (UID: "33c7a18b-c654-4401-88da-bd2875dd61c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:00.553303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.553079 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-istio-data\") pod \"352d7948-b3c7-4698-bf83-f7246a134611\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " Oct 29 23:04:00.553303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.553128 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-workload-socket\") pod \"352d7948-b3c7-4698-bf83-f7246a134611\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " Oct 29 23:04:00.553592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.553330 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-istio-envoy\") pod \"352d7948-b3c7-4698-bf83-f7246a134611\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " Oct 29 23:04:00.553592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.553371 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/352d7948-b3c7-4698-bf83-f7246a134611-istio-token\" (UniqueName: \"kubernetes.io/projected/352d7948-b3c7-4698-bf83-f7246a134611-istio-token\") pod \"352d7948-b3c7-4698-bf83-f7246a134611\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " Oct 29 23:04:00.553592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.553426 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-credential-socket\") pod \"352d7948-b3c7-4698-bf83-f7246a134611\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " Oct 29 23:04:00.553744 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.553723 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/352d7948-b3c7-4698-bf83-f7246a134611-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352d7948-b3c7-4698-bf83-f7246a134611-istiod-ca-cert\") pod \"352d7948-b3c7-4698-bf83-f7246a134611\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " Oct 29 23:04:00.553800 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.553782 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/352d7948-b3c7-4698-bf83-f7246a134611-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352d7948-b3c7-4698-bf83-f7246a134611-istio-podinfo\") pod \"352d7948-b3c7-4698-bf83-f7246a134611\" (UID: \"352d7948-b3c7-4698-bf83-f7246a134611\") " Oct 29 23:04:00.560651 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.560296 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-credential-socket" pod "352d7948-b3c7-4698-bf83-f7246a134611" (UID: "352d7948-b3c7-4698-bf83-f7246a134611"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:00.568573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.568539 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/352d7948-b3c7-4698-bf83-f7246a134611-istiod-ca-cert" pod "352d7948-b3c7-4698-bf83-f7246a134611" (UID: "352d7948-b3c7-4698-bf83-f7246a134611"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:00.569093 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.569067 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-workload-socket" pod "352d7948-b3c7-4698-bf83-f7246a134611" (UID: "352d7948-b3c7-4698-bf83-f7246a134611"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:00.581801 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.581768 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-istio-data" pod "352d7948-b3c7-4698-bf83-f7246a134611" (UID: "352d7948-b3c7-4698-bf83-f7246a134611"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:00.593278 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.593244 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-token" pod "33c7a18b-c654-4401-88da-bd2875dd61c9" (UID: "33c7a18b-c654-4401-88da-bd2875dd61c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:00.616976 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.616939 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352d7948-b3c7-4698-bf83-f7246a134611-kube-api-access-2ngsp" pod "352d7948-b3c7-4698-bf83-f7246a134611" (UID: "352d7948-b3c7-4698-bf83-f7246a134611"). InnerVolumeSpecName "kube-api-access-2ngsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:00.640212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.623972 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352d7948-b3c7-4698-bf83-f7246a134611-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:00.640212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.624020 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:00.640212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.624059 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:00.640212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.624086 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:00.640212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.624101 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:00.640212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.624115 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:00.640212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.624128 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ngsp\" (UniqueName: \"kubernetes.io/projected/352d7948-b3c7-4698-bf83-f7246a134611-kube-api-access-2ngsp\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:00.640212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.624143 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:00.640212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.624225 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:00.640212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.624240 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:00.648533 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.648490 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-podinfo" pod "33c7a18b-c654-4401-88da-bd2875dd61c9" (UID: "33c7a18b-c654-4401-88da-bd2875dd61c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:00.656337 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.648490 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c7a18b-c654-4401-88da-bd2875dd61c9-kube-api-access-ng9wq" pod "33c7a18b-c654-4401-88da-bd2875dd61c9" (UID: "33c7a18b-c654-4401-88da-bd2875dd61c9"). InnerVolumeSpecName "kube-api-access-ng9wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:00.657631 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.657602 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/352d7948-b3c7-4698-bf83-f7246a134611-istio-podinfo" pod "352d7948-b3c7-4698-bf83-f7246a134611" (UID: "352d7948-b3c7-4698-bf83-f7246a134611"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:00.677878 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.677825 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352d7948-b3c7-4698-bf83-f7246a134611-istio-token" pod "352d7948-b3c7-4698-bf83-f7246a134611" (UID: "352d7948-b3c7-4698-bf83-f7246a134611"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:00.679430 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.679368 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-istio-envoy" pod "352d7948-b3c7-4698-bf83-f7246a134611" (UID: "352d7948-b3c7-4698-bf83-f7246a134611"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:00.728176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.727951 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352d7948-b3c7-4698-bf83-f7246a134611-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:00.728176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.727991 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/352d7948-b3c7-4698-bf83-f7246a134611-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:00.728176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.728006 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng9wq\" (UniqueName: \"kubernetes.io/projected/33c7a18b-c654-4401-88da-bd2875dd61c9-kube-api-access-ng9wq\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:00.728176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.728025 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352d7948-b3c7-4698-bf83-f7246a134611-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:00.728176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:00.728041 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33c7a18b-c654-4401-88da-bd2875dd61c9-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:01.224319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:01.224289 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" Oct 29 23:04:01.224775 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:01.224312 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh" event={"ID":"33c7a18b-c654-4401-88da-bd2875dd61c9","Type":"ContainerDied","Data":"6bc1cda4b7bb31200ddd29972ef2d8ba9397e50045bbbb36f62b768ea194680d"} Oct 29 23:04:01.224775 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:01.224384 5057 scope.go:122] "RemoveContainer" containerID="05f8ed87ac01b9b4f37447b05e30fe91682bbb60886b5c6da81cb2513265c38f" Oct 29 23:04:01.226179 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:01.226154 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" event={"ID":"352d7948-b3c7-4698-bf83-f7246a134611","Type":"ContainerDied","Data":"8f9571feb4c56c213fabc9255bd42d8a503aec82527b654ad524f807d9136954"} Oct 29 23:04:01.226352 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:01.226257 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8" Oct 29 23:04:01.237936 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:01.237917 5057 scope.go:122] "RemoveContainer" containerID="0e04eea48e01c1ba168effbeb8e3e8923e7e177c6a2d9480e1f2fbd409c6871a" Oct 29 23:04:01.301071 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:01.301036 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh"] Oct 29 23:04:01.313904 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:01.313867 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-s6dxh"] Oct 29 23:04:01.340174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:01.340138 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8"] Oct 29 23:04:01.346448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:01.346420 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85kx8"] Oct 29 23:04:01.483979 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:01.483905 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33c7a18b-c654-4401-88da-bd2875dd61c9" path="/var/lib/kubelet/pods/33c7a18b-c654-4401-88da-bd2875dd61c9/volumes" Oct 29 23:04:01.484467 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:01.484452 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="352d7948-b3c7-4698-bf83-f7246a134611" path="/var/lib/kubelet/pods/352d7948-b3c7-4698-bf83-f7246a134611/volumes" Oct 29 23:04:03.944224 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:04:03.942145 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:04:06.973509 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:06.973447 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m"] Oct 29 23:04:06.974011 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:06.973718 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" podUID="6014cb97-2d6c-4ac3-b24c-e7612e0cc900" containerName="maas-api" containerID="cri-o://ccd917add18bb5d44edef0a798f569d0984ecc993ef8a95c37e1108ce601cc02" gracePeriod=30 Oct 29 23:04:07.145784 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.145745 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d"] Oct 29 23:04:07.146017 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.145989 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d" podUID="ed7a76de-9e42-485e-8034-32d15d55834b" containerName="payload-processing" containerID="cri-o://a2ebc2bd2199d832cffcf60639d66dd4a6386acd10ca13f1963c868da680fbac" gracePeriod=30 Oct 29 23:04:07.181110 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.181076 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm"] Oct 29 23:04:07.181364 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.181333 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm" podUID="7c2efb3f-9de1-42d2-9a4e-ecc08f995e07" containerName="payload-pre-processing" containerID="cri-o://a476e7560365873ccf24e1713d650dca45d4d024bd34de3fe2383917d3309878" gracePeriod=30 Oct 29 23:04:07.310427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.310398 5057 generic.go:356] "Generic (PLEG): container finished" podID="6014cb97-2d6c-4ac3-b24c-e7612e0cc900" containerID="ccd917add18bb5d44edef0a798f569d0984ecc993ef8a95c37e1108ce601cc02" exitCode=0 Oct 29 23:04:07.310556 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.310528 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" event={"ID":"6014cb97-2d6c-4ac3-b24c-e7612e0cc900","Type":"ContainerDied","Data":"ccd917add18bb5d44edef0a798f569d0984ecc993ef8a95c37e1108ce601cc02"} Oct 29 23:04:07.312844 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.312659 5057 generic.go:356] "Generic (PLEG): container finished" podID="7c2efb3f-9de1-42d2-9a4e-ecc08f995e07" containerID="a476e7560365873ccf24e1713d650dca45d4d024bd34de3fe2383917d3309878" exitCode=0 Oct 29 23:04:07.312844 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.312781 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm" event={"ID":"7c2efb3f-9de1-42d2-9a4e-ecc08f995e07","Type":"ContainerDied","Data":"a476e7560365873ccf24e1713d650dca45d4d024bd34de3fe2383917d3309878"} Oct 29 23:04:07.315057 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.315031 5057 generic.go:356] "Generic (PLEG): container finished" podID="ed7a76de-9e42-485e-8034-32d15d55834b" containerID="a2ebc2bd2199d832cffcf60639d66dd4a6386acd10ca13f1963c868da680fbac" exitCode=0 Oct 29 23:04:07.315175 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.315143 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d" event={"ID":"ed7a76de-9e42-485e-8034-32d15d55834b","Type":"ContainerDied","Data":"a2ebc2bd2199d832cffcf60639d66dd4a6386acd10ca13f1963c868da680fbac"} Oct 29 23:04:07.337364 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.337336 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" Oct 29 23:04:07.391590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.389169 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-openshift-service-ca\") pod \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\" (UID: \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\") " Oct 29 23:04:07.391590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.389287 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-maas-api-tls\") pod \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\" (UID: \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\") " Oct 29 23:04:07.391590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.389493 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-kube-api-access-hdgpn\" (UniqueName: \"kubernetes.io/projected/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-kube-api-access-hdgpn\") pod \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\" (UID: \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\") " Oct 29 23:04:07.391590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.389593 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-metrics-certs\") pod \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\" (UID: \"6014cb97-2d6c-4ac3-b24c-e7612e0cc900\") " Oct 29 23:04:07.391590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.389835 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-openshift-service-ca" pod "6014cb97-2d6c-4ac3-b24c-e7612e0cc900" (UID: "6014cb97-2d6c-4ac3-b24c-e7612e0cc900"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:07.391590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.390472 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:07.397589 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.397541 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-maas-api-tls" pod "6014cb97-2d6c-4ac3-b24c-e7612e0cc900" (UID: "6014cb97-2d6c-4ac3-b24c-e7612e0cc900"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:07.398078 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.398047 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-metrics-certs" pod "6014cb97-2d6c-4ac3-b24c-e7612e0cc900" (UID: "6014cb97-2d6c-4ac3-b24c-e7612e0cc900"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:07.400315 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.400290 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-kube-api-access-hdgpn" pod "6014cb97-2d6c-4ac3-b24c-e7612e0cc900" (UID: "6014cb97-2d6c-4ac3-b24c-e7612e0cc900"). InnerVolumeSpecName "kube-api-access-hdgpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:07.492999 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.492910 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdgpn\" (UniqueName: \"kubernetes.io/projected/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-kube-api-access-hdgpn\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:07.493248 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.493226 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:07.493406 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.493393 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6014cb97-2d6c-4ac3-b24c-e7612e0cc900-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:07.828314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.828174 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d" Oct 29 23:04:07.925616 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.925571 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed7a76de-9e42-485e-8034-32d15d55834b-kube-api-access-rzxsr\" (UniqueName: \"kubernetes.io/projected/ed7a76de-9e42-485e-8034-32d15d55834b-kube-api-access-rzxsr\") pod \"ed7a76de-9e42-485e-8034-32d15d55834b\" (UID: \"ed7a76de-9e42-485e-8034-32d15d55834b\") " Oct 29 23:04:07.926111 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.926077 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed7a76de-9e42-485e-8034-32d15d55834b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7a76de-9e42-485e-8034-32d15d55834b-plugins-config-volume\") pod \"ed7a76de-9e42-485e-8034-32d15d55834b\" (UID: \"ed7a76de-9e42-485e-8034-32d15d55834b\") " Oct 29 23:04:07.929746 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:07.928472 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7a76de-9e42-485e-8034-32d15d55834b-plugins-config-volume" pod "ed7a76de-9e42-485e-8034-32d15d55834b" (UID: "ed7a76de-9e42-485e-8034-32d15d55834b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:08.099108 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.055127 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7a76de-9e42-485e-8034-32d15d55834b-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:08.220001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.206571 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7a76de-9e42-485e-8034-32d15d55834b-kube-api-access-rzxsr" pod "ed7a76de-9e42-485e-8034-32d15d55834b" (UID: "ed7a76de-9e42-485e-8034-32d15d55834b"). InnerVolumeSpecName "kube-api-access-rzxsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:08.275435 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.275382 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzxsr\" (UniqueName: \"kubernetes.io/projected/ed7a76de-9e42-485e-8034-32d15d55834b-kube-api-access-rzxsr\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:08.517697 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.517649 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" event={"ID":"6014cb97-2d6c-4ac3-b24c-e7612e0cc900","Type":"ContainerDied","Data":"5f4c65c8a85e1631139b41ddf99bcd5c813e65552e33aa70463b9b957ca8baf6"} Oct 29 23:04:08.518306 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.518263 5057 scope.go:122] "RemoveContainer" containerID="ccd917add18bb5d44edef0a798f569d0984ecc993ef8a95c37e1108ce601cc02" Oct 29 23:04:08.518777 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.518730 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m" Oct 29 23:04:08.519580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.519557 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm" Oct 29 23:04:08.543318 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.531655 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm" event={"ID":"7c2efb3f-9de1-42d2-9a4e-ecc08f995e07","Type":"ContainerDied","Data":"3a58eae9f55542e699b20272280e2f56b5fcc3d9d80d3a6f867373bbc0631554"} Oct 29 23:04:08.552152 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.552116 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d" event={"ID":"ed7a76de-9e42-485e-8034-32d15d55834b","Type":"ContainerDied","Data":"db017ab2419b4035cd1fce58badb267f80f94e62349ef8c6bf5e9b4ea1ac00f2"} Oct 29 23:04:08.552965 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.552738 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d" Oct 29 23:04:08.577823 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.577790 5057 scope.go:122] "RemoveContainer" containerID="a476e7560365873ccf24e1713d650dca45d4d024bd34de3fe2383917d3309878" Oct 29 23:04:08.592632 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.592608 5057 scope.go:122] "RemoveContainer" containerID="a2ebc2bd2199d832cffcf60639d66dd4a6386acd10ca13f1963c868da680fbac" Oct 29 23:04:08.640798 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.640763 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c2efb3f-9de1-42d2-9a4e-ecc08f995e07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2efb3f-9de1-42d2-9a4e-ecc08f995e07-plugins-config-volume\") pod \"7c2efb3f-9de1-42d2-9a4e-ecc08f995e07\" (UID: \"7c2efb3f-9de1-42d2-9a4e-ecc08f995e07\") " Oct 29 23:04:08.640985 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.640943 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c2efb3f-9de1-42d2-9a4e-ecc08f995e07-kube-api-access-q7775\" (UniqueName: \"kubernetes.io/projected/7c2efb3f-9de1-42d2-9a4e-ecc08f995e07-kube-api-access-q7775\") pod \"7c2efb3f-9de1-42d2-9a4e-ecc08f995e07\" (UID: \"7c2efb3f-9de1-42d2-9a4e-ecc08f995e07\") " Oct 29 23:04:08.641951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.641919 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c2efb3f-9de1-42d2-9a4e-ecc08f995e07-plugins-config-volume" pod "7c2efb3f-9de1-42d2-9a4e-ecc08f995e07" (UID: "7c2efb3f-9de1-42d2-9a4e-ecc08f995e07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:08.642119 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.642059 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2efb3f-9de1-42d2-9a4e-ecc08f995e07-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:08.643962 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.643936 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2efb3f-9de1-42d2-9a4e-ecc08f995e07-kube-api-access-q7775" pod "7c2efb3f-9de1-42d2-9a4e-ecc08f995e07" (UID: "7c2efb3f-9de1-42d2-9a4e-ecc08f995e07"). InnerVolumeSpecName "kube-api-access-q7775". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:08.647375 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.644284 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d"] Oct 29 23:04:08.649775 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.649758 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-660540-b5f8b5ccb-hqq9d"] Oct 29 23:04:08.685364 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.685335 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m"] Oct 29 23:04:08.689458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.689433 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-660540-9fd475c6f-fjp5m"] Oct 29 23:04:08.743775 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:08.743742 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7775\" (UniqueName: \"kubernetes.io/projected/7c2efb3f-9de1-42d2-9a4e-ecc08f995e07-kube-api-access-q7775\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:09.483325 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:09.483298 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6014cb97-2d6c-4ac3-b24c-e7612e0cc900" path="/var/lib/kubelet/pods/6014cb97-2d6c-4ac3-b24c-e7612e0cc900/volumes" Oct 29 23:04:09.483811 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:09.483795 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed7a76de-9e42-485e-8034-32d15d55834b" path="/var/lib/kubelet/pods/ed7a76de-9e42-485e-8034-32d15d55834b/volumes" Oct 29 23:04:09.562031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:09.562001 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm" Oct 29 23:04:09.622912 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:09.622878 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm"] Oct 29 23:04:09.635486 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:09.635451 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-76cdcdb69f-gbccm"] Oct 29 23:04:10.194429 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.194393 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9"] Oct 29 23:04:10.195346 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195321 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9f124749-58f1-491c-b3e8-f1a0d0adb8dc" containerName="istio-proxy" Oct 29 23:04:10.195346 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195344 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f124749-58f1-491c-b3e8-f1a0d0adb8dc" containerName="istio-proxy" Oct 29 23:04:10.195655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195365 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7c2efb3f-9de1-42d2-9a4e-ecc08f995e07" containerName="payload-pre-processing" Oct 29 23:04:10.195655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195375 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c2efb3f-9de1-42d2-9a4e-ecc08f995e07" containerName="payload-pre-processing" Oct 29 23:04:10.195655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195435 5057 cpu_manager.go:414] "containerMap: removing container" podUID="299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0" containerName="payload-pre-processing" Oct 29 23:04:10.195655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195444 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0" containerName="payload-pre-processing" Oct 29 23:04:10.195655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195459 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9f124749-58f1-491c-b3e8-f1a0d0adb8dc" containerName="istio-proxy" Oct 29 23:04:10.195655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195466 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f124749-58f1-491c-b3e8-f1a0d0adb8dc" containerName="istio-proxy" Oct 29 23:04:10.195655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195498 5057 cpu_manager.go:414] "containerMap: removing container" podUID="ed7a76de-9e42-485e-8034-32d15d55834b" containerName="payload-processing" Oct 29 23:04:10.195655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195532 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed7a76de-9e42-485e-8034-32d15d55834b" containerName="payload-processing" Oct 29 23:04:10.195655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195558 5057 cpu_manager.go:414] "containerMap: removing container" podUID="352d7948-b3c7-4698-bf83-f7246a134611" containerName="istio-proxy" Oct 29 23:04:10.195655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195564 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="352d7948-b3c7-4698-bf83-f7246a134611" containerName="istio-proxy" Oct 29 23:04:10.195655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195576 5057 cpu_manager.go:414] "containerMap: removing container" podUID="6014cb97-2d6c-4ac3-b24c-e7612e0cc900" containerName="maas-api" Oct 29 23:04:10.195655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195583 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6014cb97-2d6c-4ac3-b24c-e7612e0cc900" containerName="maas-api" Oct 29 23:04:10.195655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195599 5057 cpu_manager.go:414] "containerMap: removing container" podUID="33c7a18b-c654-4401-88da-bd2875dd61c9" containerName="istio-proxy" Oct 29 23:04:10.195655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195606 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c7a18b-c654-4401-88da-bd2875dd61c9" containerName="istio-proxy" Oct 29 23:04:10.195655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195616 5057 cpu_manager.go:414] "containerMap: removing container" podUID="97751900-4954-4a2a-bf49-99b702e4a7c2" containerName="payload-processing" Oct 29 23:04:10.195655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195624 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97751900-4954-4a2a-bf49-99b702e4a7c2" containerName="payload-processing" Oct 29 23:04:10.196469 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195829 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="299b8fd2-97f2-4faf-b7a2-0e2fbbb064b0" containerName="payload-pre-processing" Oct 29 23:04:10.196469 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195842 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f124749-58f1-491c-b3e8-f1a0d0adb8dc" containerName="istio-proxy" Oct 29 23:04:10.196469 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195858 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed7a76de-9e42-485e-8034-32d15d55834b" containerName="payload-processing" Oct 29 23:04:10.196469 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195871 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="352d7948-b3c7-4698-bf83-f7246a134611" containerName="istio-proxy" Oct 29 23:04:10.196469 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195881 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="6014cb97-2d6c-4ac3-b24c-e7612e0cc900" containerName="maas-api" Oct 29 23:04:10.196469 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195892 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c7a18b-c654-4401-88da-bd2875dd61c9" containerName="istio-proxy" Oct 29 23:04:10.196469 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195900 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="97751900-4954-4a2a-bf49-99b702e4a7c2" containerName="payload-processing" Oct 29 23:04:10.196469 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.195917 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c2efb3f-9de1-42d2-9a4e-ecc08f995e07" containerName="payload-pre-processing" Oct 29 23:04:10.200419 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.200396 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.226350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.226319 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9"] Oct 29 23:04:10.235089 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.234144 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-41b12c-openshift-default-dockercfg-dw6v5\"" Oct 29 23:04:10.272357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.272317 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-workload-certs\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.272583 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.272489 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztpvs\" (UniqueName: \"kubernetes.io/projected/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-kube-api-access-ztpvs\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.272662 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.272630 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-envoy\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.272725 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.272689 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-podinfo\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.272785 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.272721 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-data\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.272785 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.272767 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-credential-socket\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.272899 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.272802 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-token\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.273014 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.272968 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istiod-ca-cert\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.273338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.273306 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-workload-socket\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.401210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.385707 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-workload-socket\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.401210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.385916 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-workload-certs\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.401210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.386052 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztpvs\" (UniqueName: \"kubernetes.io/projected/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-kube-api-access-ztpvs\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.401210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.386144 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-envoy\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.401210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.386220 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-podinfo\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.429213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.401730 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-data\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.429213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.401834 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-credential-socket\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.429213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.401876 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-token\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.429213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.401909 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istiod-ca-cert\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.429213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.402959 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istiod-ca-cert\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.429213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.403270 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-workload-socket\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.429213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.415484 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-workload-certs\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.429213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.419499 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-data\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.429213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.422512 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-credential-socket\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.478052 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.477974 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-podinfo\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.498217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.494131 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-token\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.506506 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.506463 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-envoy\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.551904 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.551867 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztpvs\" (UniqueName: \"kubernetes.io/projected/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-kube-api-access-ztpvs\") pod \"e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:10.822768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:10.821530 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:11.451152 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:04:11.451122 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9c9cf6_5558_4e14_a1c1_1ca61f40e3c1.slice/crio-12ec086bbdf7e391a124000d5ca38f40982ea4325f150f0fc27d9e9c71e7c756 WatchSource:0}: Error finding container 12ec086bbdf7e391a124000d5ca38f40982ea4325f150f0fc27d9e9c71e7c756: Status 404 returned error can't find the container with id 12ec086bbdf7e391a124000d5ca38f40982ea4325f150f0fc27d9e9c71e7c756 Oct 29 23:04:11.451970 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:11.451936 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9"] Oct 29 23:04:11.454479 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:11.454405 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:04:11.454570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:11.454523 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:04:11.454625 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:11.454582 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:04:11.486264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:11.486238 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c2efb3f-9de1-42d2-9a4e-ecc08f995e07" path="/var/lib/kubelet/pods/7c2efb3f-9de1-42d2-9a4e-ecc08f995e07/volumes" Oct 29 23:04:11.644788 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:11.644650 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" event={"ID":"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1","Type":"ContainerStarted","Data":"e4e8a3003c3c34e7083d941ab9221ee7feefdcae06a8aefa92c99f5038b1a4ff"} Oct 29 23:04:11.644788 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:11.644745 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" event={"ID":"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1","Type":"ContainerStarted","Data":"12ec086bbdf7e391a124000d5ca38f40982ea4325f150f0fc27d9e9c71e7c756"} Oct 29 23:04:11.697553 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:11.697450 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" podStartSLOduration=1.697433338 podStartE2EDuration="1.697433338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:11.693867001 +0000 UTC m=+1298.345365531" watchObservedRunningTime="2026-10-29 23:04:11.697433338 +0000 UTC m=+1298.348931868" Oct 29 23:04:11.838213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:11.831305 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:12.850824 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:12.850479 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:13.768252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:13.767850 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:13.786214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:13.780540 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:04:16.222111 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:16.222014 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj"] Oct 29 23:04:16.222689 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:16.222317 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" podUID="a2f1b565-3ffb-4859-9284-024f97d2f84a" containerName="istio-proxy" containerID="cri-o://d371450bc265a3437b4cd04b19a6d8e8feaf6fd9ef4358d61203996e3a2e17f6" gracePeriod=30 Oct 29 23:04:16.272704 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:16.272666 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk"] Oct 29 23:04:16.272894 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:16.272871 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" podUID="4e7c1f33-f809-4677-b63d-1d8ad1a36551" containerName="istio-proxy" containerID="cri-o://aaac47fc6c1abd57a240f46aa1cd3446a3c7a6c221694f04f9f0352793622ca9" gracePeriod=30 Oct 29 23:04:16.279445 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:16.279423 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf"] Oct 29 23:04:16.279736 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:16.279715 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" podUID="17a46d30-1423-40eb-b6ec-00345bfd910a" containerName="istio-proxy" containerID="cri-o://75eeba9b629322a33e1a8527186808554ac25ce56d172d229e8df79e27edb3af" gracePeriod=30 Oct 29 23:04:17.209613 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:17.209572 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl"] Oct 29 23:04:17.210633 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:17.210610 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f124749-58f1-491c-b3e8-f1a0d0adb8dc" containerName="istio-proxy" Oct 29 23:04:17.214095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:17.214076 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" Oct 29 23:04:17.242572 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:17.242547 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b0c72c-serving-cert\"" Oct 29 23:04:17.270290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:17.270247 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl"] Oct 29 23:04:17.304636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:17.304602 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-openshift-service-ca\") pod \"maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl\" (UID: \"7d0a6630-350e-4f6d-9c7f-c261cc545ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" Oct 29 23:04:17.304815 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:17.304668 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-maas-api-tls\") pod \"maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl\" (UID: \"7d0a6630-350e-4f6d-9c7f-c261cc545ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" Oct 29 23:04:17.304934 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:17.304916 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-metrics-certs\") pod \"maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl\" (UID: \"7d0a6630-350e-4f6d-9c7f-c261cc545ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" Oct 29 23:04:17.304974 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:17.304946 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n2sk\" (UniqueName: \"kubernetes.io/projected/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-kube-api-access-8n2sk\") pod \"maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl\" (UID: \"7d0a6630-350e-4f6d-9c7f-c261cc545ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" Oct 29 23:04:17.406783 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:17.406744 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-metrics-certs\") pod \"maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl\" (UID: \"7d0a6630-350e-4f6d-9c7f-c261cc545ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" Oct 29 23:04:17.406783 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:17.406784 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n2sk\" (UniqueName: \"kubernetes.io/projected/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-kube-api-access-8n2sk\") pod \"maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl\" (UID: \"7d0a6630-350e-4f6d-9c7f-c261cc545ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" Oct 29 23:04:17.407035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:17.406886 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-openshift-service-ca\") pod \"maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl\" (UID: \"7d0a6630-350e-4f6d-9c7f-c261cc545ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" Oct 29 23:04:17.407035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:17.406924 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-maas-api-tls\") pod \"maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl\" (UID: \"7d0a6630-350e-4f6d-9c7f-c261cc545ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" Oct 29 23:04:17.407625 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:17.407592 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-openshift-service-ca\") pod \"maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl\" (UID: \"7d0a6630-350e-4f6d-9c7f-c261cc545ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" Oct 29 23:04:17.409176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:17.409145 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-metrics-certs\") pod \"maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl\" (UID: \"7d0a6630-350e-4f6d-9c7f-c261cc545ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" Oct 29 23:04:17.409471 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:17.409450 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-maas-api-tls\") pod \"maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl\" (UID: \"7d0a6630-350e-4f6d-9c7f-c261cc545ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" Oct 29 23:04:17.436548 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:17.436519 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n2sk\" (UniqueName: \"kubernetes.io/projected/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-kube-api-access-8n2sk\") pod \"maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl\" (UID: \"7d0a6630-350e-4f6d-9c7f-c261cc545ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" Oct 29 23:04:17.533478 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:17.532595 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" Oct 29 23:04:17.754304 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:17.752785 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:04:17.754304 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:17.752912 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" podUID="a2f1b565-3ffb-4859-9284-024f97d2f84a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:04:18.031147 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:04:18.031111 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0a6630_350e_4f6d_9c7f_c261cc545ea5.slice/crio-d49bb51727c03833559b638593974f320ed99122c2bd755b8c693d387c424982 WatchSource:0}: Error finding container d49bb51727c03833559b638593974f320ed99122c2bd755b8c693d387c424982: Status 404 returned error can't find the container with id d49bb51727c03833559b638593974f320ed99122c2bd755b8c693d387c424982 Oct 29 23:04:18.032105 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:18.032080 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl"] Oct 29 23:04:18.032844 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:18.032826 5057 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:04:18.145230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:18.145115 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:04:18.145544 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:18.145503 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" podUID="17a46d30-1423-40eb-b6ec-00345bfd910a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:04:18.838265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:18.838177 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" event={"ID":"7d0a6630-350e-4f6d-9c7f-c261cc545ea5","Type":"ContainerStarted","Data":"c5298ae8cb8b7fc1caf9bd2578f0796b8386381dca8d8ac29b7ee1bb4cc17f2e"} Oct 29 23:04:18.838265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:18.838267 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" Oct 29 23:04:18.838816 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:18.838290 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" event={"ID":"7d0a6630-350e-4f6d-9c7f-c261cc545ea5","Type":"ContainerStarted","Data":"d49bb51727c03833559b638593974f320ed99122c2bd755b8c693d387c424982"} Oct 29 23:04:18.885582 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:18.885529 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" podStartSLOduration=1.540414555 podStartE2EDuration="1.885515317s" totalImagesPullingTime="345.100762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:18.884034483 +0000 UTC m=+1305.535533012" watchObservedRunningTime="2026-10-29 23:04:18.885515317 +0000 UTC m=+1305.537013847" Oct 29 23:04:21.621161 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.621129 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:04:21.699037 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.698955 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:04:21.701523 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.701455 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:04:21.728755 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.728720 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-data\") pod \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " Oct 29 23:04:21.729470 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.729444 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-credential-socket\") pod \"a2f1b565-3ffb-4859-9284-024f97d2f84a\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " Oct 29 23:04:21.729918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.729900 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a46d30-1423-40eb-b6ec-00345bfd910a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a46d30-1423-40eb-b6ec-00345bfd910a-istiod-ca-cert\") pod \"17a46d30-1423-40eb-b6ec-00345bfd910a\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " Oct 29 23:04:21.730052 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.729221 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-data" pod "4e7c1f33-f809-4677-b63d-1d8ad1a36551" (UID: "4e7c1f33-f809-4677-b63d-1d8ad1a36551"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.730549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.729828 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-credential-socket" pod "a2f1b565-3ffb-4859-9284-024f97d2f84a" (UID: "a2f1b565-3ffb-4859-9284-024f97d2f84a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.730654 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.730500 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a46d30-1423-40eb-b6ec-00345bfd910a-istiod-ca-cert" pod "17a46d30-1423-40eb-b6ec-00345bfd910a" (UID: "17a46d30-1423-40eb-b6ec-00345bfd910a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:21.730654 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.730505 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istiod-ca-cert" pod "4e7c1f33-f809-4677-b63d-1d8ad1a36551" (UID: "4e7c1f33-f809-4677-b63d-1d8ad1a36551"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:21.730654 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.730128 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istiod-ca-cert\") pod \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " Oct 29 23:04:21.730654 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.730637 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a46d30-1423-40eb-b6ec-00345bfd910a-kube-api-access-v222t\" (UniqueName: \"kubernetes.io/projected/17a46d30-1423-40eb-b6ec-00345bfd910a-kube-api-access-v222t\") pod \"17a46d30-1423-40eb-b6ec-00345bfd910a\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " Oct 29 23:04:21.730876 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.730678 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-token\" (UniqueName: \"kubernetes.io/projected/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-token\") pod \"a2f1b565-3ffb-4859-9284-024f97d2f84a\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " Oct 29 23:04:21.730876 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.730723 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-workload-certs\") pod \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " Oct 29 23:04:21.730876 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.730767 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7c1f33-f809-4677-b63d-1d8ad1a36551-kube-api-access-mq9p2\" (UniqueName: \"kubernetes.io/projected/4e7c1f33-f809-4677-b63d-1d8ad1a36551-kube-api-access-mq9p2\") pod \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " Oct 29 23:04:21.730876 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.730800 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f1b565-3ffb-4859-9284-024f97d2f84a-kube-api-access-vhvf4\" (UniqueName: \"kubernetes.io/projected/a2f1b565-3ffb-4859-9284-024f97d2f84a-kube-api-access-vhvf4\") pod \"a2f1b565-3ffb-4859-9284-024f97d2f84a\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " Oct 29 23:04:21.730876 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.730855 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-podinfo\") pod \"a2f1b565-3ffb-4859-9284-024f97d2f84a\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " Oct 29 23:04:21.731119 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.730891 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-workload-socket\") pod \"a2f1b565-3ffb-4859-9284-024f97d2f84a\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " Oct 29 23:04:21.731119 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.730918 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-token\" (UniqueName: \"kubernetes.io/projected/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-token\") pod \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " Oct 29 23:04:21.731119 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.730985 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-credential-socket\") pod \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " Oct 29 23:04:21.731119 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.731025 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-podinfo\") pod \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " Oct 29 23:04:21.731119 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.731110 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-workload-socket\") pod \"17a46d30-1423-40eb-b6ec-00345bfd910a\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " Oct 29 23:04:21.733168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.731146 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-workload-socket\") pod \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " Oct 29 23:04:21.733168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.731227 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2f1b565-3ffb-4859-9284-024f97d2f84a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f1b565-3ffb-4859-9284-024f97d2f84a-istiod-ca-cert\") pod \"a2f1b565-3ffb-4859-9284-024f97d2f84a\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " Oct 29 23:04:21.733168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.731282 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-envoy\") pod \"17a46d30-1423-40eb-b6ec-00345bfd910a\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " Oct 29 23:04:21.733168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.731346 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-envoy\") pod \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\" (UID: \"4e7c1f33-f809-4677-b63d-1d8ad1a36551\") " Oct 29 23:04:21.733168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.731396 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-podinfo\") pod \"17a46d30-1423-40eb-b6ec-00345bfd910a\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " Oct 29 23:04:21.733168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.731444 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-envoy\") pod \"a2f1b565-3ffb-4859-9284-024f97d2f84a\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " Oct 29 23:04:21.733168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.731474 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-workload-certs\") pod \"a2f1b565-3ffb-4859-9284-024f97d2f84a\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " Oct 29 23:04:21.733168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.731505 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-token\" (UniqueName: \"kubernetes.io/projected/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-token\") pod \"17a46d30-1423-40eb-b6ec-00345bfd910a\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " Oct 29 23:04:21.733168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.731572 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-workload-certs\") pod \"17a46d30-1423-40eb-b6ec-00345bfd910a\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " Oct 29 23:04:21.733168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.731627 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-data\") pod \"a2f1b565-3ffb-4859-9284-024f97d2f84a\" (UID: \"a2f1b565-3ffb-4859-9284-024f97d2f84a\") " Oct 29 23:04:21.733168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.731655 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-data\") pod \"17a46d30-1423-40eb-b6ec-00345bfd910a\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " Oct 29 23:04:21.733168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.731705 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-credential-socket\") pod \"17a46d30-1423-40eb-b6ec-00345bfd910a\" (UID: \"17a46d30-1423-40eb-b6ec-00345bfd910a\") " Oct 29 23:04:21.733168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.732714 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-workload-socket" pod "17a46d30-1423-40eb-b6ec-00345bfd910a" (UID: "17a46d30-1423-40eb-b6ec-00345bfd910a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.733168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.732897 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-credential-socket" pod "17a46d30-1423-40eb-b6ec-00345bfd910a" (UID: "17a46d30-1423-40eb-b6ec-00345bfd910a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.733168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.733074 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-workload-socket" pod "4e7c1f33-f809-4677-b63d-1d8ad1a36551" (UID: "4e7c1f33-f809-4677-b63d-1d8ad1a36551"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.737344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.733403 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f1b565-3ffb-4859-9284-024f97d2f84a-istiod-ca-cert" pod "a2f1b565-3ffb-4859-9284-024f97d2f84a" (UID: "a2f1b565-3ffb-4859-9284-024f97d2f84a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:21.737344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.733872 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.737344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.733896 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.737344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.733911 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.737344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.733924 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a46d30-1423-40eb-b6ec-00345bfd910a-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.737344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.733938 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.737344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.733954 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.737344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.733966 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.737344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.733981 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f1b565-3ffb-4859-9284-024f97d2f84a-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.737344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.734063 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-workload-certs" pod "a2f1b565-3ffb-4859-9284-024f97d2f84a" (UID: "a2f1b565-3ffb-4859-9284-024f97d2f84a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.737344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.734925 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-workload-certs" pod "17a46d30-1423-40eb-b6ec-00345bfd910a" (UID: "17a46d30-1423-40eb-b6ec-00345bfd910a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.737344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.735887 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-data" pod "a2f1b565-3ffb-4859-9284-024f97d2f84a" (UID: "a2f1b565-3ffb-4859-9284-024f97d2f84a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.737344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.736112 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-workload-socket" pod "a2f1b565-3ffb-4859-9284-024f97d2f84a" (UID: "a2f1b565-3ffb-4859-9284-024f97d2f84a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.737344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.736153 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-data" pod "17a46d30-1423-40eb-b6ec-00345bfd910a" (UID: "17a46d30-1423-40eb-b6ec-00345bfd910a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.737344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.736408 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-credential-socket" pod "4e7c1f33-f809-4677-b63d-1d8ad1a36551" (UID: "4e7c1f33-f809-4677-b63d-1d8ad1a36551"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.739625 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.739427 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-envoy" pod "a2f1b565-3ffb-4859-9284-024f97d2f84a" (UID: "a2f1b565-3ffb-4859-9284-024f97d2f84a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.739684 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.739638 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-workload-certs" pod "4e7c1f33-f809-4677-b63d-1d8ad1a36551" (UID: "4e7c1f33-f809-4677-b63d-1d8ad1a36551"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.747799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.747767 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-podinfo" pod "4e7c1f33-f809-4677-b63d-1d8ad1a36551" (UID: "4e7c1f33-f809-4677-b63d-1d8ad1a36551"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:21.747977 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.747954 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-token" pod "4e7c1f33-f809-4677-b63d-1d8ad1a36551" (UID: "4e7c1f33-f809-4677-b63d-1d8ad1a36551"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:21.748115 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.748090 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7c1f33-f809-4677-b63d-1d8ad1a36551-kube-api-access-mq9p2" pod "4e7c1f33-f809-4677-b63d-1d8ad1a36551" (UID: "4e7c1f33-f809-4677-b63d-1d8ad1a36551"). InnerVolumeSpecName "kube-api-access-mq9p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:21.748569 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.748249 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-token" pod "17a46d30-1423-40eb-b6ec-00345bfd910a" (UID: "17a46d30-1423-40eb-b6ec-00345bfd910a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:21.748569 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.748306 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-envoy" pod "4e7c1f33-f809-4677-b63d-1d8ad1a36551" (UID: "4e7c1f33-f809-4677-b63d-1d8ad1a36551"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.748569 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.748343 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-podinfo" pod "17a46d30-1423-40eb-b6ec-00345bfd910a" (UID: "17a46d30-1423-40eb-b6ec-00345bfd910a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:21.748569 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.748337 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-podinfo" pod "a2f1b565-3ffb-4859-9284-024f97d2f84a" (UID: "a2f1b565-3ffb-4859-9284-024f97d2f84a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:21.748569 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.748469 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a46d30-1423-40eb-b6ec-00345bfd910a-kube-api-access-v222t" pod "17a46d30-1423-40eb-b6ec-00345bfd910a" (UID: "17a46d30-1423-40eb-b6ec-00345bfd910a"). InnerVolumeSpecName "kube-api-access-v222t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:21.748569 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.748523 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-token" pod "a2f1b565-3ffb-4859-9284-024f97d2f84a" (UID: "a2f1b565-3ffb-4859-9284-024f97d2f84a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:21.748569 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.748533 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-envoy" pod "17a46d30-1423-40eb-b6ec-00345bfd910a" (UID: "17a46d30-1423-40eb-b6ec-00345bfd910a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.750238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.750165 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f1b565-3ffb-4859-9284-024f97d2f84a-kube-api-access-vhvf4" pod "a2f1b565-3ffb-4859-9284-024f97d2f84a" (UID: "a2f1b565-3ffb-4859-9284-024f97d2f84a"). InnerVolumeSpecName "kube-api-access-vhvf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:21.795794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.795703 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr"] Oct 29 23:04:21.797101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.797074 5057 cpu_manager.go:414] "containerMap: removing container" podUID="17a46d30-1423-40eb-b6ec-00345bfd910a" containerName="istio-proxy" Oct 29 23:04:21.797101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.797099 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a46d30-1423-40eb-b6ec-00345bfd910a" containerName="istio-proxy" Oct 29 23:04:21.797373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.797161 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a2f1b565-3ffb-4859-9284-024f97d2f84a" containerName="istio-proxy" Oct 29 23:04:21.797373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.797169 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f1b565-3ffb-4859-9284-024f97d2f84a" containerName="istio-proxy" Oct 29 23:04:21.797373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.797221 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4e7c1f33-f809-4677-b63d-1d8ad1a36551" containerName="istio-proxy" Oct 29 23:04:21.797373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.797229 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7c1f33-f809-4677-b63d-1d8ad1a36551" containerName="istio-proxy" Oct 29 23:04:21.797596 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.797412 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a46d30-1423-40eb-b6ec-00345bfd910a" containerName="istio-proxy" Oct 29 23:04:21.797596 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.797438 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f1b565-3ffb-4859-9284-024f97d2f84a" containerName="istio-proxy" Oct 29 23:04:21.797596 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.797452 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7c1f33-f809-4677-b63d-1d8ad1a36551" containerName="istio-proxy" Oct 29 23:04:21.803784 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.803761 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.808225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.807952 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn"] Oct 29 23:04:21.812485 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.812463 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.827982 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.827945 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr"] Oct 29 23:04:21.840648 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.840501 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24nbs\" (UniqueName: \"kubernetes.io/projected/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-kube-api-access-24nbs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.840998 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.840975 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b619e2-1f43-4b7e-98e8-b83587057e45-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.841170 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.841151 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.841319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.841302 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.841441 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.841425 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.841627 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.841610 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.841785 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.841768 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.841917 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.841903 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.842021 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.842007 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.842135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.842120 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.842389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.842373 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.842585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.842571 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.842723 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.842708 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4gjf\" (UniqueName: \"kubernetes.io/projected/93b619e2-1f43-4b7e-98e8-b83587057e45-kube-api-access-c4gjf\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.854619 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.854582 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.854650 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.854698 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.854727 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.854752 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.855026 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.855053 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.855068 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.855082 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.855095 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.855641 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.855661 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.855673 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.855686 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a46d30-1423-40eb-b6ec-00345bfd910a-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.855702 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v222t\" (UniqueName: \"kubernetes.io/projected/17a46d30-1423-40eb-b6ec-00345bfd910a-kube-api-access-v222t\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.855716 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.855733 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.855746 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq9p2\" (UniqueName: \"kubernetes.io/projected/4e7c1f33-f809-4677-b63d-1d8ad1a36551-kube-api-access-mq9p2\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.855759 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhvf4\" (UniqueName: \"kubernetes.io/projected/a2f1b565-3ffb-4859-9284-024f97d2f84a-kube-api-access-vhvf4\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.855771 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f1b565-3ffb-4859-9284-024f97d2f84a-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.855787 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f1b565-3ffb-4859-9284-024f97d2f84a-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.855799 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.855813 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7c1f33-f809-4677-b63d-1d8ad1a36551-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.855865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.855828 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7c1f33-f809-4677-b63d-1d8ad1a36551-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.878517 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.878483 5057 generic.go:356] "Generic (PLEG): container finished" podID="4e7c1f33-f809-4677-b63d-1d8ad1a36551" containerID="aaac47fc6c1abd57a240f46aa1cd3446a3c7a6c221694f04f9f0352793622ca9" exitCode=0 Oct 29 23:04:21.878833 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.878553 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" event={"ID":"4e7c1f33-f809-4677-b63d-1d8ad1a36551","Type":"ContainerDied","Data":"aaac47fc6c1abd57a240f46aa1cd3446a3c7a6c221694f04f9f0352793622ca9"} Oct 29 23:04:21.879467 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.878594 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" Oct 29 23:04:21.879575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.879468 5057 scope.go:122] "RemoveContainer" containerID="aaac47fc6c1abd57a240f46aa1cd3446a3c7a6c221694f04f9f0352793622ca9" Oct 29 23:04:21.879575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.879449 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk" event={"ID":"4e7c1f33-f809-4677-b63d-1d8ad1a36551","Type":"ContainerDied","Data":"7b27f3a77a780c7f1b7f8788652fc8c812238d066e4c38dca24664750e087a89"} Oct 29 23:04:21.881909 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.881515 5057 generic.go:356] "Generic (PLEG): container finished" podID="a2f1b565-3ffb-4859-9284-024f97d2f84a" containerID="d371450bc265a3437b4cd04b19a6d8e8feaf6fd9ef4358d61203996e3a2e17f6" exitCode=0 Oct 29 23:04:21.881909 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.881622 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" event={"ID":"a2f1b565-3ffb-4859-9284-024f97d2f84a","Type":"ContainerDied","Data":"d371450bc265a3437b4cd04b19a6d8e8feaf6fd9ef4358d61203996e3a2e17f6"} Oct 29 23:04:21.881909 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.881658 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" event={"ID":"a2f1b565-3ffb-4859-9284-024f97d2f84a","Type":"ContainerDied","Data":"8db1ee082b733202ea97e529d1c6fac81689e2e6812f7f27ddd09632350da1a2"} Oct 29 23:04:21.881909 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.881735 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj" Oct 29 23:04:21.882904 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.882881 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn"] Oct 29 23:04:21.885781 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.885752 5057 generic.go:356] "Generic (PLEG): container finished" podID="17a46d30-1423-40eb-b6ec-00345bfd910a" containerID="75eeba9b629322a33e1a8527186808554ac25ce56d172d229e8df79e27edb3af" exitCode=0 Oct 29 23:04:21.885951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.885933 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" event={"ID":"17a46d30-1423-40eb-b6ec-00345bfd910a","Type":"ContainerDied","Data":"75eeba9b629322a33e1a8527186808554ac25ce56d172d229e8df79e27edb3af"} Oct 29 23:04:21.886034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.885978 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" event={"ID":"17a46d30-1423-40eb-b6ec-00345bfd910a","Type":"ContainerDied","Data":"7f22317126c86c1a539368bf1a7b6b1275da6d7a41c57393158adaab8e1e63df"} Oct 29 23:04:21.886113 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.886097 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf" Oct 29 23:04:21.897723 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.897348 5057 scope.go:122] "RemoveContainer" containerID="aaac47fc6c1abd57a240f46aa1cd3446a3c7a6c221694f04f9f0352793622ca9" Oct 29 23:04:21.898181 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:04:21.898029 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaac47fc6c1abd57a240f46aa1cd3446a3c7a6c221694f04f9f0352793622ca9\": container with ID starting with aaac47fc6c1abd57a240f46aa1cd3446a3c7a6c221694f04f9f0352793622ca9 not found: ID does not exist" containerID="aaac47fc6c1abd57a240f46aa1cd3446a3c7a6c221694f04f9f0352793622ca9" Oct 29 23:04:21.898181 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.898068 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaac47fc6c1abd57a240f46aa1cd3446a3c7a6c221694f04f9f0352793622ca9"} err="failed to get container status \"aaac47fc6c1abd57a240f46aa1cd3446a3c7a6c221694f04f9f0352793622ca9\": rpc error: code = NotFound desc = could not find container \"aaac47fc6c1abd57a240f46aa1cd3446a3c7a6c221694f04f9f0352793622ca9\": container with ID starting with aaac47fc6c1abd57a240f46aa1cd3446a3c7a6c221694f04f9f0352793622ca9 not found: ID does not exist" Oct 29 23:04:21.898181 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.898092 5057 scope.go:122] "RemoveContainer" containerID="d371450bc265a3437b4cd04b19a6d8e8feaf6fd9ef4358d61203996e3a2e17f6" Oct 29 23:04:21.924598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.924576 5057 scope.go:122] "RemoveContainer" containerID="d371450bc265a3437b4cd04b19a6d8e8feaf6fd9ef4358d61203996e3a2e17f6" Oct 29 23:04:21.925230 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:04:21.925197 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d371450bc265a3437b4cd04b19a6d8e8feaf6fd9ef4358d61203996e3a2e17f6\": container with ID starting with d371450bc265a3437b4cd04b19a6d8e8feaf6fd9ef4358d61203996e3a2e17f6 not found: ID does not exist" containerID="d371450bc265a3437b4cd04b19a6d8e8feaf6fd9ef4358d61203996e3a2e17f6" Oct 29 23:04:21.925367 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.925236 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d371450bc265a3437b4cd04b19a6d8e8feaf6fd9ef4358d61203996e3a2e17f6"} err="failed to get container status \"d371450bc265a3437b4cd04b19a6d8e8feaf6fd9ef4358d61203996e3a2e17f6\": rpc error: code = NotFound desc = could not find container \"d371450bc265a3437b4cd04b19a6d8e8feaf6fd9ef4358d61203996e3a2e17f6\": container with ID starting with d371450bc265a3437b4cd04b19a6d8e8feaf6fd9ef4358d61203996e3a2e17f6 not found: ID does not exist" Oct 29 23:04:21.925367 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.925259 5057 scope.go:122] "RemoveContainer" containerID="75eeba9b629322a33e1a8527186808554ac25ce56d172d229e8df79e27edb3af" Oct 29 23:04:21.938177 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.938157 5057 scope.go:122] "RemoveContainer" containerID="75eeba9b629322a33e1a8527186808554ac25ce56d172d229e8df79e27edb3af" Oct 29 23:04:21.938669 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:04:21.938642 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75eeba9b629322a33e1a8527186808554ac25ce56d172d229e8df79e27edb3af\": container with ID starting with 75eeba9b629322a33e1a8527186808554ac25ce56d172d229e8df79e27edb3af not found: ID does not exist" containerID="75eeba9b629322a33e1a8527186808554ac25ce56d172d229e8df79e27edb3af" Oct 29 23:04:21.938759 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.938679 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75eeba9b629322a33e1a8527186808554ac25ce56d172d229e8df79e27edb3af"} err="failed to get container status \"75eeba9b629322a33e1a8527186808554ac25ce56d172d229e8df79e27edb3af\": rpc error: code = NotFound desc = could not find container \"75eeba9b629322a33e1a8527186808554ac25ce56d172d229e8df79e27edb3af\": container with ID starting with 75eeba9b629322a33e1a8527186808554ac25ce56d172d229e8df79e27edb3af not found: ID does not exist" Oct 29 23:04:21.957409 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.957375 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.957568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.957435 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.957568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.957479 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.957568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.957511 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.957568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.957537 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.957804 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.957584 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24nbs\" (UniqueName: \"kubernetes.io/projected/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-kube-api-access-24nbs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.957804 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.957616 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b619e2-1f43-4b7e-98e8-b83587057e45-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.957804 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.957656 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.957804 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.957681 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.957804 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.957713 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.957804 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.957747 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.958096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.957828 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.958096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.957881 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.958096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.957913 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.958096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.957930 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.958096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.957951 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.958096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.958028 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.958096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.958079 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.958096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.958096 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4gjf\" (UniqueName: \"kubernetes.io/projected/93b619e2-1f43-4b7e-98e8-b83587057e45-kube-api-access-c4gjf\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.958558 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.958108 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.958558 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.958445 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.959244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.959175 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b619e2-1f43-4b7e-98e8-b83587057e45-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.959814 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.959443 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.959814 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.959712 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.960824 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.960080 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.960824 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.960182 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.960824 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.960252 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.960824 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.960363 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.961855 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.961823 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.962105 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.962082 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.963469 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.963448 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.964073 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.964051 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.985870 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.985846 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.986023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.985995 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24nbs\" (UniqueName: \"kubernetes.io/projected/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-kube-api-access-24nbs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:21.986023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.985916 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.986220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.985946 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4gjf\" (UniqueName: \"kubernetes.io/projected/93b619e2-1f43-4b7e-98e8-b83587057e45-kube-api-access-c4gjf\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:21.992465 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.992437 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf"] Oct 29 23:04:21.999131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:21.999050 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9mktf"] Oct 29 23:04:22.032930 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:22.032887 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj"] Oct 29 23:04:22.142424 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:22.137206 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:22.162987 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:22.155415 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-sc4xj"] Oct 29 23:04:22.165344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:22.165316 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk"] Oct 29 23:04:22.165566 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:22.165545 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-9h4dk"] Oct 29 23:04:22.176598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:22.176163 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:22.533126 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:22.533056 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr"] Oct 29 23:04:22.543714 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:22.543667 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:04:22.543864 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:22.543765 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:04:22.543864 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:22.543807 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:04:22.592261 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:04:22.592233 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b619e2_1f43_4b7e_98e8_b83587057e45.slice/crio-26cde0e9e232a90a5a26ca9fec04e0bc5272cbcf70563010b18b6aa8de06b468 WatchSource:0}: Error finding container 26cde0e9e232a90a5a26ca9fec04e0bc5272cbcf70563010b18b6aa8de06b468: Status 404 returned error can't find the container with id 26cde0e9e232a90a5a26ca9fec04e0bc5272cbcf70563010b18b6aa8de06b468 Oct 29 23:04:22.595752 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:22.595728 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn"] Oct 29 23:04:22.596117 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:22.596085 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:04:22.596235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:22.596158 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:04:22.596235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:22.596209 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:04:22.902009 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:22.901914 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" event={"ID":"93b619e2-1f43-4b7e-98e8-b83587057e45","Type":"ContainerStarted","Data":"7fb315758b2f371c0a427df37e500c0f438a1dca21eebf34d335c9f0ce7ea557"} Oct 29 23:04:22.902639 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:22.902609 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" event={"ID":"93b619e2-1f43-4b7e-98e8-b83587057e45","Type":"ContainerStarted","Data":"26cde0e9e232a90a5a26ca9fec04e0bc5272cbcf70563010b18b6aa8de06b468"} Oct 29 23:04:22.909254 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:22.909163 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" event={"ID":"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b","Type":"ContainerStarted","Data":"1dda5a8541a8a3d9c471f86ab24f64041d10795edfde8582cbb1943188ee4347"} Oct 29 23:04:22.909254 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:22.909239 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" event={"ID":"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b","Type":"ContainerStarted","Data":"dafe07869c8e2a215ff03e1b687471d60ecaa9824f881c2febb8411586d8df41"} Oct 29 23:04:22.957293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:22.957172 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" podStartSLOduration=1.9571522460000002 podStartE2EDuration="1.957152246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:22.949834637 +0000 UTC m=+1309.601333170" watchObservedRunningTime="2026-10-29 23:04:22.957152246 +0000 UTC m=+1309.608650777" Oct 29 23:04:23.002329 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:23.002279 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" podStartSLOduration=2.00226449 podStartE2EDuration="2.00226449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:22.998076498 +0000 UTC m=+1309.649575029" watchObservedRunningTime="2026-10-29 23:04:23.00226449 +0000 UTC m=+1309.653763010" Oct 29 23:04:23.157394 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:23.157290 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:23.159020 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:23.158980 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 29 23:04:23.159208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:23.159059 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" podUID="e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 29 23:04:23.181990 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:23.181952 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:23.191611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:23.191446 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 29 23:04:23.191611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:23.191512 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" podUID="93b619e2-1f43-4b7e-98e8-b83587057e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 29 23:04:23.490368 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:23.490338 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17a46d30-1423-40eb-b6ec-00345bfd910a" path="/var/lib/kubelet/pods/17a46d30-1423-40eb-b6ec-00345bfd910a/volumes" Oct 29 23:04:23.490874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:23.490859 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e7c1f33-f809-4677-b63d-1d8ad1a36551" path="/var/lib/kubelet/pods/4e7c1f33-f809-4677-b63d-1d8ad1a36551/volumes" Oct 29 23:04:23.491357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:23.491343 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f1b565-3ffb-4859-9284-024f97d2f84a" path="/var/lib/kubelet/pods/a2f1b565-3ffb-4859-9284-024f97d2f84a/volumes" Oct 29 23:04:24.163213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:24.160483 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 29 23:04:24.163213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:24.160547 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" podUID="e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 29 23:04:24.187211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:24.181599 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 29 23:04:24.187211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:24.181658 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" podUID="93b619e2-1f43-4b7e-98e8-b83587057e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 29 23:04:24.883615 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:24.883585 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" Oct 29 23:04:25.168651 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:25.168559 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 29 23:04:25.169228 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:25.169179 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" podUID="e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 29 23:04:25.179739 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:25.179632 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 29 23:04:25.179739 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:25.179677 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" podUID="93b619e2-1f43-4b7e-98e8-b83587057e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 29 23:04:26.160630 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:26.160589 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 29 23:04:26.160940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:26.160910 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" podUID="e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 29 23:04:26.178642 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:26.178609 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 29 23:04:26.179177 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:26.179149 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" podUID="93b619e2-1f43-4b7e-98e8-b83587057e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 29 23:04:27.176242 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:27.162761 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 29 23:04:27.176242 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:27.162839 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" podUID="e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 29 23:04:27.189221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:27.179824 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 29 23:04:27.189830 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:27.189773 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" podUID="93b619e2-1f43-4b7e-98e8-b83587057e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 29 23:04:28.161040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:28.160584 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 29 23:04:28.161040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:28.160664 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" podUID="e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 29 23:04:28.179208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:28.179076 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 29 23:04:28.179208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:28.179144 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" podUID="93b619e2-1f43-4b7e-98e8-b83587057e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 29 23:04:29.163098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:29.163067 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:29.164318 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:29.164291 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:29.165243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:29.165215 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:04:29.182861 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:29.182836 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:29.183157 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:29.183137 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:29.183912 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:29.183892 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:04:30.433839 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:30.433801 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd"] Oct 29 23:04:30.434382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:30.434009 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd" podUID="7241fa67-618b-4720-a365-9bf989b92131" containerName="payload-processing" containerID="cri-o://232aa9d95f0177adf59df279d4e3c46a9949e3435ec837fc31eb669b1cecfb98" gracePeriod=30 Oct 29 23:04:30.485036 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:30.485002 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq"] Oct 29 23:04:30.485514 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:30.485422 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq" podUID="a06ccf8d-5d83-474f-8528-e900ee22f687" containerName="payload-pre-processing" containerID="cri-o://e64fe5c37c4bac65f119fc8f8fffa5703b59bc3374711e3dcb20c47b93867ed4" gracePeriod=30 Oct 29 23:04:30.869254 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:30.866024 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd" Oct 29 23:04:30.871008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:30.870972 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq" Oct 29 23:04:30.990112 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:30.990023 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7241fa67-618b-4720-a365-9bf989b92131-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7241fa67-618b-4720-a365-9bf989b92131-plugins-config-volume\") pod \"7241fa67-618b-4720-a365-9bf989b92131\" (UID: \"7241fa67-618b-4720-a365-9bf989b92131\") " Oct 29 23:04:30.990314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:30.990289 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a06ccf8d-5d83-474f-8528-e900ee22f687-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06ccf8d-5d83-474f-8528-e900ee22f687-plugins-config-volume\") pod \"a06ccf8d-5d83-474f-8528-e900ee22f687\" (UID: \"a06ccf8d-5d83-474f-8528-e900ee22f687\") " Oct 29 23:04:30.990430 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:30.990404 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7241fa67-618b-4720-a365-9bf989b92131-kube-api-access-9r2qx\" (UniqueName: \"kubernetes.io/projected/7241fa67-618b-4720-a365-9bf989b92131-kube-api-access-9r2qx\") pod \"7241fa67-618b-4720-a365-9bf989b92131\" (UID: \"7241fa67-618b-4720-a365-9bf989b92131\") " Oct 29 23:04:30.990504 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:30.990417 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7241fa67-618b-4720-a365-9bf989b92131-plugins-config-volume" pod "7241fa67-618b-4720-a365-9bf989b92131" (UID: "7241fa67-618b-4720-a365-9bf989b92131"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:30.990504 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:30.990475 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a06ccf8d-5d83-474f-8528-e900ee22f687-kube-api-access-s9jmh\" (UniqueName: \"kubernetes.io/projected/a06ccf8d-5d83-474f-8528-e900ee22f687-kube-api-access-s9jmh\") pod \"a06ccf8d-5d83-474f-8528-e900ee22f687\" (UID: \"a06ccf8d-5d83-474f-8528-e900ee22f687\") " Oct 29 23:04:30.990695 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:30.990670 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06ccf8d-5d83-474f-8528-e900ee22f687-plugins-config-volume" pod "a06ccf8d-5d83-474f-8528-e900ee22f687" (UID: "a06ccf8d-5d83-474f-8528-e900ee22f687"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:30.991348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:30.991323 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7241fa67-618b-4720-a365-9bf989b92131-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:30.991473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:30.991352 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06ccf8d-5d83-474f-8528-e900ee22f687-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:30.992716 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:30.992688 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7241fa67-618b-4720-a365-9bf989b92131-kube-api-access-9r2qx" pod "7241fa67-618b-4720-a365-9bf989b92131" (UID: "7241fa67-618b-4720-a365-9bf989b92131"). InnerVolumeSpecName "kube-api-access-9r2qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:30.993182 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:30.993151 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06ccf8d-5d83-474f-8528-e900ee22f687-kube-api-access-s9jmh" pod "a06ccf8d-5d83-474f-8528-e900ee22f687" (UID: "a06ccf8d-5d83-474f-8528-e900ee22f687"). InnerVolumeSpecName "kube-api-access-s9jmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:31.096211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.094432 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r2qx\" (UniqueName: \"kubernetes.io/projected/7241fa67-618b-4720-a365-9bf989b92131-kube-api-access-9r2qx\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:31.096211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.094467 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9jmh\" (UniqueName: \"kubernetes.io/projected/a06ccf8d-5d83-474f-8528-e900ee22f687-kube-api-access-s9jmh\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:31.213793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.208325 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk"] Oct 29 23:04:31.213793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.208632 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" podUID="078c4461-ea44-4c8c-af2e-ae1e44776e79" containerName="maas-api" containerID="cri-o://f24471f62e56fafc44e36717f9a7073e9af37f5523d1ffdf0fa760652ef16357" gracePeriod=30 Oct 29 23:04:31.739250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.737884 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk"] Oct 29 23:04:31.739250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.738446 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk" podUID="356ee5cf-6fec-47b4-90c1-0c1354b74c18" containerName="payload-processing" containerID="cri-o://165218c229ef1c8fd64e58950c0b18841dd2b6336a2c97174e0f1850aabde700" gracePeriod=30 Oct 29 23:04:31.759895 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.759855 5057 generic.go:356] "Generic (PLEG): container finished" podID="7241fa67-618b-4720-a365-9bf989b92131" containerID="232aa9d95f0177adf59df279d4e3c46a9949e3435ec837fc31eb669b1cecfb98" exitCode=0 Oct 29 23:04:31.760584 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.760559 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd" Oct 29 23:04:31.768147 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.768103 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd" event={"ID":"7241fa67-618b-4720-a365-9bf989b92131","Type":"ContainerDied","Data":"232aa9d95f0177adf59df279d4e3c46a9949e3435ec837fc31eb669b1cecfb98"} Oct 29 23:04:31.768283 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.768198 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd" event={"ID":"7241fa67-618b-4720-a365-9bf989b92131","Type":"ContainerDied","Data":"bb8260049284ddda0e31b646ab9bd35275ad30fbcef0536e0eed4d0ed13517dc"} Oct 29 23:04:31.768353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.768325 5057 scope.go:122] "RemoveContainer" containerID="232aa9d95f0177adf59df279d4e3c46a9949e3435ec837fc31eb669b1cecfb98" Oct 29 23:04:31.775441 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.775414 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq"] Oct 29 23:04:31.783385 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.782350 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq" podUID="a06ccf8d-5d83-474f-8528-e900ee22f687" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.132:9004: i/o timeout" Oct 29 23:04:31.783385 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.783302 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq" podUID="b49892fa-80a0-4f7b-a75d-fb027e183254" containerName="payload-pre-processing" containerID="cri-o://6521468e891f10a4226c91c6d742b40f9deabd014d28b5620750730d00971815" gracePeriod=30 Oct 29 23:04:31.787222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.787177 5057 generic.go:356] "Generic (PLEG): container finished" podID="a06ccf8d-5d83-474f-8528-e900ee22f687" containerID="e64fe5c37c4bac65f119fc8f8fffa5703b59bc3374711e3dcb20c47b93867ed4" exitCode=0 Oct 29 23:04:31.787429 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.787389 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq" event={"ID":"a06ccf8d-5d83-474f-8528-e900ee22f687","Type":"ContainerDied","Data":"e64fe5c37c4bac65f119fc8f8fffa5703b59bc3374711e3dcb20c47b93867ed4"} Oct 29 23:04:31.787521 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.787463 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq" event={"ID":"a06ccf8d-5d83-474f-8528-e900ee22f687","Type":"ContainerDied","Data":"1cb91cf276256ba608d229ada29e5a8bc352b948a2b6bf6d1e8c288541653f11"} Oct 29 23:04:31.787623 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.787603 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq" Oct 29 23:04:31.874383 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.867274 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd"] Oct 29 23:04:31.874383 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.874062 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-475876-5c98b9b5d4-sddvd"] Oct 29 23:04:31.966857 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.966722 5057 scope.go:122] "RemoveContainer" containerID="232aa9d95f0177adf59df279d4e3c46a9949e3435ec837fc31eb669b1cecfb98" Oct 29 23:04:31.967978 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:04:31.967788 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"232aa9d95f0177adf59df279d4e3c46a9949e3435ec837fc31eb669b1cecfb98\": container with ID starting with 232aa9d95f0177adf59df279d4e3c46a9949e3435ec837fc31eb669b1cecfb98 not found: ID does not exist" containerID="232aa9d95f0177adf59df279d4e3c46a9949e3435ec837fc31eb669b1cecfb98" Oct 29 23:04:31.967978 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.967830 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232aa9d95f0177adf59df279d4e3c46a9949e3435ec837fc31eb669b1cecfb98"} err="failed to get container status \"232aa9d95f0177adf59df279d4e3c46a9949e3435ec837fc31eb669b1cecfb98\": rpc error: code = NotFound desc = could not find container \"232aa9d95f0177adf59df279d4e3c46a9949e3435ec837fc31eb669b1cecfb98\": container with ID starting with 232aa9d95f0177adf59df279d4e3c46a9949e3435ec837fc31eb669b1cecfb98 not found: ID does not exist" Oct 29 23:04:31.967978 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.967856 5057 scope.go:122] "RemoveContainer" containerID="e64fe5c37c4bac65f119fc8f8fffa5703b59bc3374711e3dcb20c47b93867ed4" Oct 29 23:04:31.999951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:31.999923 5057 scope.go:122] "RemoveContainer" containerID="e64fe5c37c4bac65f119fc8f8fffa5703b59bc3374711e3dcb20c47b93867ed4" Oct 29 23:04:32.000469 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:04:32.000382 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e64fe5c37c4bac65f119fc8f8fffa5703b59bc3374711e3dcb20c47b93867ed4\": container with ID starting with e64fe5c37c4bac65f119fc8f8fffa5703b59bc3374711e3dcb20c47b93867ed4 not found: ID does not exist" containerID="e64fe5c37c4bac65f119fc8f8fffa5703b59bc3374711e3dcb20c47b93867ed4" Oct 29 23:04:32.000469 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.000420 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e64fe5c37c4bac65f119fc8f8fffa5703b59bc3374711e3dcb20c47b93867ed4"} err="failed to get container status \"e64fe5c37c4bac65f119fc8f8fffa5703b59bc3374711e3dcb20c47b93867ed4\": rpc error: code = NotFound desc = could not find container \"e64fe5c37c4bac65f119fc8f8fffa5703b59bc3374711e3dcb20c47b93867ed4\": container with ID starting with e64fe5c37c4bac65f119fc8f8fffa5703b59bc3374711e3dcb20c47b93867ed4 not found: ID does not exist" Oct 29 23:04:32.010681 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.010624 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq"] Oct 29 23:04:32.017786 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.017749 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-475876-85497c85bc-kxtwq"] Oct 29 23:04:32.674866 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.672404 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" Oct 29 23:04:32.786497 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.786348 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/078c4461-ea44-4c8c-af2e-ae1e44776e79-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/078c4461-ea44-4c8c-af2e-ae1e44776e79-openshift-service-ca\") pod \"078c4461-ea44-4c8c-af2e-ae1e44776e79\" (UID: \"078c4461-ea44-4c8c-af2e-ae1e44776e79\") " Oct 29 23:04:32.787245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.787141 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/078c4461-ea44-4c8c-af2e-ae1e44776e79-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078c4461-ea44-4c8c-af2e-ae1e44776e79-maas-api-tls\") pod \"078c4461-ea44-4c8c-af2e-ae1e44776e79\" (UID: \"078c4461-ea44-4c8c-af2e-ae1e44776e79\") " Oct 29 23:04:32.787451 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.787431 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/078c4461-ea44-4c8c-af2e-ae1e44776e79-kube-api-access-zqv4f\" (UniqueName: \"kubernetes.io/projected/078c4461-ea44-4c8c-af2e-ae1e44776e79-kube-api-access-zqv4f\") pod \"078c4461-ea44-4c8c-af2e-ae1e44776e79\" (UID: \"078c4461-ea44-4c8c-af2e-ae1e44776e79\") " Oct 29 23:04:32.787647 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.787610 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/078c4461-ea44-4c8c-af2e-ae1e44776e79-metrics-certs\" (UniqueName: \"kubernetes.io/secret/078c4461-ea44-4c8c-af2e-ae1e44776e79-metrics-certs\") pod \"078c4461-ea44-4c8c-af2e-ae1e44776e79\" (UID: \"078c4461-ea44-4c8c-af2e-ae1e44776e79\") " Oct 29 23:04:32.796749 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.794898 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078c4461-ea44-4c8c-af2e-ae1e44776e79-openshift-service-ca" pod "078c4461-ea44-4c8c-af2e-ae1e44776e79" (UID: "078c4461-ea44-4c8c-af2e-ae1e44776e79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:32.803615 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.803589 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078c4461-ea44-4c8c-af2e-ae1e44776e79-maas-api-tls" pod "078c4461-ea44-4c8c-af2e-ae1e44776e79" (UID: "078c4461-ea44-4c8c-af2e-ae1e44776e79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:32.805029 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.804990 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078c4461-ea44-4c8c-af2e-ae1e44776e79-kube-api-access-zqv4f" pod "078c4461-ea44-4c8c-af2e-ae1e44776e79" (UID: "078c4461-ea44-4c8c-af2e-ae1e44776e79"). InnerVolumeSpecName "kube-api-access-zqv4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:32.818020 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.817865 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078c4461-ea44-4c8c-af2e-ae1e44776e79-metrics-certs" pod "078c4461-ea44-4c8c-af2e-ae1e44776e79" (UID: "078c4461-ea44-4c8c-af2e-ae1e44776e79"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:32.856481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.856448 5057 generic.go:356] "Generic (PLEG): container finished" podID="356ee5cf-6fec-47b4-90c1-0c1354b74c18" containerID="165218c229ef1c8fd64e58950c0b18841dd2b6336a2c97174e0f1850aabde700" exitCode=0 Oct 29 23:04:32.856839 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.856815 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk" event={"ID":"356ee5cf-6fec-47b4-90c1-0c1354b74c18","Type":"ContainerDied","Data":"165218c229ef1c8fd64e58950c0b18841dd2b6336a2c97174e0f1850aabde700"} Oct 29 23:04:32.858862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.858838 5057 generic.go:356] "Generic (PLEG): container finished" podID="b49892fa-80a0-4f7b-a75d-fb027e183254" containerID="6521468e891f10a4226c91c6d742b40f9deabd014d28b5620750730d00971815" exitCode=0 Oct 29 23:04:32.859077 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.859062 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq" event={"ID":"b49892fa-80a0-4f7b-a75d-fb027e183254","Type":"ContainerDied","Data":"6521468e891f10a4226c91c6d742b40f9deabd014d28b5620750730d00971815"} Oct 29 23:04:32.860383 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.860362 5057 generic.go:356] "Generic (PLEG): container finished" podID="078c4461-ea44-4c8c-af2e-ae1e44776e79" containerID="f24471f62e56fafc44e36717f9a7073e9af37f5523d1ffdf0fa760652ef16357" exitCode=0 Oct 29 23:04:32.860563 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.860417 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" Oct 29 23:04:32.860802 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.860431 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" event={"ID":"078c4461-ea44-4c8c-af2e-ae1e44776e79","Type":"ContainerDied","Data":"f24471f62e56fafc44e36717f9a7073e9af37f5523d1ffdf0fa760652ef16357"} Oct 29 23:04:32.860939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.860924 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" event={"ID":"078c4461-ea44-4c8c-af2e-ae1e44776e79","Type":"ContainerDied","Data":"9a4a253c9b7e53d7863a5120688c292821cd87a8de0f56cdf8f9c167b77161d8"} Oct 29 23:04:32.861054 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.861041 5057 scope.go:122] "RemoveContainer" containerID="f24471f62e56fafc44e36717f9a7073e9af37f5523d1ffdf0fa760652ef16357" Oct 29 23:04:32.890090 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.890068 5057 scope.go:122] "RemoveContainer" containerID="f24471f62e56fafc44e36717f9a7073e9af37f5523d1ffdf0fa760652ef16357" Oct 29 23:04:32.890286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.890261 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078c4461-ea44-4c8c-af2e-ae1e44776e79-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:32.890381 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.890290 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqv4f\" (UniqueName: \"kubernetes.io/projected/078c4461-ea44-4c8c-af2e-ae1e44776e79-kube-api-access-zqv4f\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:32.890381 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.890307 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078c4461-ea44-4c8c-af2e-ae1e44776e79-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:32.890381 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.890324 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/078c4461-ea44-4c8c-af2e-ae1e44776e79-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:32.890504 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:04:32.890416 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f24471f62e56fafc44e36717f9a7073e9af37f5523d1ffdf0fa760652ef16357\": container with ID starting with f24471f62e56fafc44e36717f9a7073e9af37f5523d1ffdf0fa760652ef16357 not found: ID does not exist" containerID="f24471f62e56fafc44e36717f9a7073e9af37f5523d1ffdf0fa760652ef16357" Oct 29 23:04:32.890504 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.890444 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f24471f62e56fafc44e36717f9a7073e9af37f5523d1ffdf0fa760652ef16357"} err="failed to get container status \"f24471f62e56fafc44e36717f9a7073e9af37f5523d1ffdf0fa760652ef16357\": rpc error: code = NotFound desc = could not find container \"f24471f62e56fafc44e36717f9a7073e9af37f5523d1ffdf0fa760652ef16357\": container with ID starting with f24471f62e56fafc44e36717f9a7073e9af37f5523d1ffdf0fa760652ef16357 not found: ID does not exist" Oct 29 23:04:32.891233 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.891213 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq" Oct 29 23:04:32.894514 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.894497 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk" Oct 29 23:04:32.931698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.931656 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk"] Oct 29 23:04:32.938306 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.938280 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk"] Oct 29 23:04:32.990895 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.990858 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/356ee5cf-6fec-47b4-90c1-0c1354b74c18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356ee5cf-6fec-47b4-90c1-0c1354b74c18-plugins-config-volume\") pod \"356ee5cf-6fec-47b4-90c1-0c1354b74c18\" (UID: \"356ee5cf-6fec-47b4-90c1-0c1354b74c18\") " Oct 29 23:04:32.991848 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:32.991808 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49892fa-80a0-4f7b-a75d-fb027e183254-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b49892fa-80a0-4f7b-a75d-fb027e183254-plugins-config-volume\") pod \"b49892fa-80a0-4f7b-a75d-fb027e183254\" (UID: \"b49892fa-80a0-4f7b-a75d-fb027e183254\") " Oct 29 23:04:33.002706 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.002665 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49892fa-80a0-4f7b-a75d-fb027e183254-kube-api-access-llxp2\" (UniqueName: \"kubernetes.io/projected/b49892fa-80a0-4f7b-a75d-fb027e183254-kube-api-access-llxp2\") pod \"b49892fa-80a0-4f7b-a75d-fb027e183254\" (UID: \"b49892fa-80a0-4f7b-a75d-fb027e183254\") " Oct 29 23:04:33.002997 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.002825 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/356ee5cf-6fec-47b4-90c1-0c1354b74c18-kube-api-access-jn2hh\" (UniqueName: \"kubernetes.io/projected/356ee5cf-6fec-47b4-90c1-0c1354b74c18-kube-api-access-jn2hh\") pod \"356ee5cf-6fec-47b4-90c1-0c1354b74c18\" (UID: \"356ee5cf-6fec-47b4-90c1-0c1354b74c18\") " Oct 29 23:04:33.012972 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.002482 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49892fa-80a0-4f7b-a75d-fb027e183254-plugins-config-volume" pod "b49892fa-80a0-4f7b-a75d-fb027e183254" (UID: "b49892fa-80a0-4f7b-a75d-fb027e183254"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:33.016629 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.016572 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49892fa-80a0-4f7b-a75d-fb027e183254-kube-api-access-llxp2" pod "b49892fa-80a0-4f7b-a75d-fb027e183254" (UID: "b49892fa-80a0-4f7b-a75d-fb027e183254"). InnerVolumeSpecName "kube-api-access-llxp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:33.016797 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.016714 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/356ee5cf-6fec-47b4-90c1-0c1354b74c18-kube-api-access-jn2hh" pod "356ee5cf-6fec-47b4-90c1-0c1354b74c18" (UID: "356ee5cf-6fec-47b4-90c1-0c1354b74c18"). InnerVolumeSpecName "kube-api-access-jn2hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:33.125764 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.119999 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b49892fa-80a0-4f7b-a75d-fb027e183254-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.125764 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.120037 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llxp2\" (UniqueName: \"kubernetes.io/projected/b49892fa-80a0-4f7b-a75d-fb027e183254-kube-api-access-llxp2\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.125764 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.120054 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn2hh\" (UniqueName: \"kubernetes.io/projected/356ee5cf-6fec-47b4-90c1-0c1354b74c18-kube-api-access-jn2hh\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.125764 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.120942 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/356ee5cf-6fec-47b4-90c1-0c1354b74c18-plugins-config-volume" pod "356ee5cf-6fec-47b4-90c1-0c1354b74c18" (UID: "356ee5cf-6fec-47b4-90c1-0c1354b74c18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:33.221290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.221255 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356ee5cf-6fec-47b4-90c1-0c1354b74c18-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.484157 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.484126 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="078c4461-ea44-4c8c-af2e-ae1e44776e79" path="/var/lib/kubelet/pods/078c4461-ea44-4c8c-af2e-ae1e44776e79/volumes" Oct 29 23:04:33.484636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.484621 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7241fa67-618b-4720-a365-9bf989b92131" path="/var/lib/kubelet/pods/7241fa67-618b-4720-a365-9bf989b92131/volumes" Oct 29 23:04:33.485049 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.485035 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a06ccf8d-5d83-474f-8528-e900ee22f687" path="/var/lib/kubelet/pods/a06ccf8d-5d83-474f-8528-e900ee22f687/volumes" Oct 29 23:04:33.878690 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.878610 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk" Oct 29 23:04:33.878690 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.878633 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk" event={"ID":"356ee5cf-6fec-47b4-90c1-0c1354b74c18","Type":"ContainerDied","Data":"8b2e7f0210e7c2cf8c0f8dc9bc3a922b9be3ba7d3c6bbf55cdf1d0dfb22079a0"} Oct 29 23:04:33.878690 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.878693 5057 scope.go:122] "RemoveContainer" containerID="165218c229ef1c8fd64e58950c0b18841dd2b6336a2c97174e0f1850aabde700" Oct 29 23:04:33.880449 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.880424 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq" Oct 29 23:04:33.880586 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.880427 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq" event={"ID":"b49892fa-80a0-4f7b-a75d-fb027e183254","Type":"ContainerDied","Data":"921b3e8fa06a2a1c71d7c5438d31f2201d8ed0acbba376a60d4f0a2dfea6842a"} Oct 29 23:04:33.890517 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.890498 5057 scope.go:122] "RemoveContainer" containerID="6521468e891f10a4226c91c6d742b40f9deabd014d28b5620750730d00971815" Oct 29 23:04:33.918140 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:04:33.918106 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:04:33.938144 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.938097 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq"] Oct 29 23:04:33.951706 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.951679 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-74f6486c59-ntnkq"] Oct 29 23:04:33.989589 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.989561 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk"] Oct 29 23:04:33.995468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:33.995444 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-50734e-5f77555dd8-tztmk"] Oct 29 23:04:34.976271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:34.973366 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-50734e-67bb65ddb9-rw7mk" podUID="078c4461-ea44-4c8c-af2e-ae1e44776e79" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.137:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:04:35.483325 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:35.483294 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="356ee5cf-6fec-47b4-90c1-0c1354b74c18" path="/var/lib/kubelet/pods/356ee5cf-6fec-47b4-90c1-0c1354b74c18/volumes" Oct 29 23:04:35.483731 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:35.483718 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49892fa-80a0-4f7b-a75d-fb027e183254" path="/var/lib/kubelet/pods/b49892fa-80a0-4f7b-a75d-fb027e183254/volumes" Oct 29 23:04:35.524867 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:35.524837 5057 scope.go:122] "RemoveContainer" containerID="9ca5bb60d789b5c89f9c856897cc5fccb615ee21def3a3745f409bbf16e0d48e" Oct 29 23:04:37.027303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:36.985628 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt"] Oct 29 23:04:37.027303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:36.986931 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7241fa67-618b-4720-a365-9bf989b92131" containerName="payload-processing" Oct 29 23:04:37.027303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:36.986954 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7241fa67-618b-4720-a365-9bf989b92131" containerName="payload-processing" Oct 29 23:04:37.027303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:36.987008 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b49892fa-80a0-4f7b-a75d-fb027e183254" containerName="payload-pre-processing" Oct 29 23:04:37.027303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:36.987018 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49892fa-80a0-4f7b-a75d-fb027e183254" containerName="payload-pre-processing" Oct 29 23:04:37.027303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:36.987080 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a06ccf8d-5d83-474f-8528-e900ee22f687" containerName="payload-pre-processing" Oct 29 23:04:37.027303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:36.987089 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a06ccf8d-5d83-474f-8528-e900ee22f687" containerName="payload-pre-processing" Oct 29 23:04:37.027303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:36.987123 5057 cpu_manager.go:414] "containerMap: removing container" podUID="356ee5cf-6fec-47b4-90c1-0c1354b74c18" containerName="payload-processing" Oct 29 23:04:37.027303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:36.987131 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="356ee5cf-6fec-47b4-90c1-0c1354b74c18" containerName="payload-processing" Oct 29 23:04:37.027303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:36.987148 5057 cpu_manager.go:414] "containerMap: removing container" podUID="078c4461-ea44-4c8c-af2e-ae1e44776e79" containerName="maas-api" Oct 29 23:04:37.027303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:36.987155 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078c4461-ea44-4c8c-af2e-ae1e44776e79" containerName="maas-api" Oct 29 23:04:37.027303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:36.987362 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49892fa-80a0-4f7b-a75d-fb027e183254" containerName="payload-pre-processing" Oct 29 23:04:37.027303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:36.987383 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a06ccf8d-5d83-474f-8528-e900ee22f687" containerName="payload-pre-processing" Oct 29 23:04:37.027303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:36.987398 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="356ee5cf-6fec-47b4-90c1-0c1354b74c18" containerName="payload-processing" Oct 29 23:04:37.027303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:36.987407 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="078c4461-ea44-4c8c-af2e-ae1e44776e79" containerName="maas-api" Oct 29 23:04:37.027303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:36.987419 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7241fa67-618b-4720-a365-9bf989b92131" containerName="payload-processing" Oct 29 23:04:37.052211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.041007 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt"] Oct 29 23:04:37.053461 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.043343 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.191570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.191533 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.191747 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.191682 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6g8q\" (UniqueName: \"kubernetes.io/projected/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-kube-api-access-r6g8q\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.191747 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.191717 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.191907 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.191885 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.191970 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.191932 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.192010 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.192000 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.192062 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.192025 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.192062 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.192045 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.192137 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.192090 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.294379 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.294298 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.294529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.294479 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.294594 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.294580 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6g8q\" (UniqueName: \"kubernetes.io/projected/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-kube-api-access-r6g8q\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.294659 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.294610 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.294749 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.294732 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.294833 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.294793 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.294898 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.294845 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.294898 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.294880 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.294995 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.294910 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.294995 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.294951 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.295125 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.295083 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.295223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.295154 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.295411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.295388 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.295493 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.295432 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.297091 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.297067 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.297757 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.297736 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.320914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.320874 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6g8q\" (UniqueName: \"kubernetes.io/projected/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-kube-api-access-r6g8q\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.321072 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.320879 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.395964 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.395927 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:37.751550 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.751517 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt"] Oct 29 23:04:37.752943 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:04:37.752906 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcda00c4_c4da_43c2_b232_3f7c877c3ef1.slice/crio-50217c638ee4d4aa908489875d7e05c9c4f428071a0c33dfca4afd8e2af7ac8c WatchSource:0}: Error finding container 50217c638ee4d4aa908489875d7e05c9c4f428071a0c33dfca4afd8e2af7ac8c: Status 404 returned error can't find the container with id 50217c638ee4d4aa908489875d7e05c9c4f428071a0c33dfca4afd8e2af7ac8c Oct 29 23:04:37.758295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.758238 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:04:37.758422 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.758403 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:04:37.758514 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:37.758495 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:04:38.141608 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:38.141500 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" event={"ID":"dcda00c4-c4da-43c2-b232-3f7c877c3ef1","Type":"ContainerStarted","Data":"45375d4ca47662b5ce77543914f8d008765ac9af51ef60cf3b15195900806c0d"} Oct 29 23:04:38.142176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:38.142151 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" event={"ID":"dcda00c4-c4da-43c2-b232-3f7c877c3ef1","Type":"ContainerStarted","Data":"50217c638ee4d4aa908489875d7e05c9c4f428071a0c33dfca4afd8e2af7ac8c"} Oct 29 23:04:38.395856 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:38.395755 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" podStartSLOduration=2.395740074 podStartE2EDuration="2.395740074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:38.230538235 +0000 UTC m=+1324.882036766" watchObservedRunningTime="2026-10-29 23:04:38.395740074 +0000 UTC m=+1325.047238603" Oct 29 23:04:38.399291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:38.398921 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 29 23:04:38.399291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:38.398972 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" podUID="dcda00c4-c4da-43c2-b232-3f7c877c3ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 29 23:04:38.399291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:38.399231 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:38.402029 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:38.402005 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb"] Oct 29 23:04:38.402309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:38.402280 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" podUID="860644cd-397b-4af8-b425-a81491c9b66b" containerName="istio-proxy" containerID="cri-o://ab9c0e9c18b3005f83902c673219e92108a47242d93426f7163492bca609c233" gracePeriod=30 Oct 29 23:04:39.397412 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:39.397373 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 29 23:04:39.397908 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:39.397431 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" podUID="dcda00c4-c4da-43c2-b232-3f7c877c3ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 29 23:04:39.903208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:39.900678 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd"] Oct 29 23:04:39.903208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:39.900924 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" podUID="9864c587-073a-4b1e-9e7c-68fe058b3c16" containerName="istio-proxy" containerID="cri-o://d300568f26076396d8067fcd3f755a541b30555047ed82d19994391a1e43440a" gracePeriod=30 Oct 29 23:04:39.914861 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:39.906655 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc"] Oct 29 23:04:39.914861 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:39.906887 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" podUID="bbfc0179-11dd-431d-af1a-0793c48ba6fe" containerName="istio-proxy" containerID="cri-o://cdc998dc73604bdd420660d1c09d263e9d4eb611a87e24c91785b95080835cbf" gracePeriod=30 Oct 29 23:04:40.476023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:40.453754 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 29 23:04:40.476023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:40.453811 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" podUID="dcda00c4-c4da-43c2-b232-3f7c877c3ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 29 23:04:40.503336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:40.496796 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:04:40.503336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:40.496855 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" podUID="860644cd-397b-4af8-b425-a81491c9b66b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:04:40.781828 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:40.781740 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq"] Oct 29 23:04:40.782024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:40.781993 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" podUID="7cf5e014-85ba-4024-9ebc-088735098c5c" containerName="istio-proxy" containerID="cri-o://d53ce11747772ebcbc2daaefe98ce68370a0c18afac031ad86a492a97d69dee9" gracePeriod=30 Oct 29 23:04:41.397319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:41.397284 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 29 23:04:41.397572 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:41.397364 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" podUID="dcda00c4-c4da-43c2-b232-3f7c877c3ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 29 23:04:42.396382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.396350 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 29 23:04:42.396750 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.396403 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" podUID="dcda00c4-c4da-43c2-b232-3f7c877c3ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 29 23:04:42.766314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.766276 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t"] Oct 29 23:04:42.777789 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.777758 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:42.796334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.796303 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-0941c8-openshift-default-dockercfg-7kbgh\"" Oct 29 23:04:42.815598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.815378 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-credential-socket\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:42.815598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.815441 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-data\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:42.815844 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.815632 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c26371-4cdf-4bf1-95d7-855d452655c3-istiod-ca-cert\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:42.815844 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.815669 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-workload-certs\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:42.815844 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.815742 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78p4d\" (UniqueName: \"kubernetes.io/projected/06c26371-4cdf-4bf1-95d7-855d452655c3-kube-api-access-78p4d\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:42.816018 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.815853 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-workload-socket\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:42.816018 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.815883 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-token\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:42.816018 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.815909 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-envoy\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:42.816018 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.815932 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-podinfo\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:42.819382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.819353 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t"] Oct 29 23:04:42.957213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.946839 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-workload-socket\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:42.957213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.946894 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-token\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:42.957213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.946921 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-envoy\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:42.957213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.946946 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-podinfo\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:42.957213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.947132 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-credential-socket\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:42.957213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.947172 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-data\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:42.957213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.947334 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c26371-4cdf-4bf1-95d7-855d452655c3-istiod-ca-cert\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:42.957213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.947376 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-workload-certs\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:42.957213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:42.947461 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78p4d\" (UniqueName: \"kubernetes.io/projected/06c26371-4cdf-4bf1-95d7-855d452655c3-kube-api-access-78p4d\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:43.226003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.180897 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-workload-socket\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:43.400208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.396938 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 29 23:04:43.400208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.397003 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" podUID="dcda00c4-c4da-43c2-b232-3f7c877c3ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 29 23:04:43.445874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.445837 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-credential-socket\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:43.447001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.446971 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c26371-4cdf-4bf1-95d7-855d452655c3-istiod-ca-cert\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:43.447692 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.447659 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-data\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:43.492441 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.463489 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-token\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:43.492441 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.466452 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78p4d\" (UniqueName: \"kubernetes.io/projected/06c26371-4cdf-4bf1-95d7-855d452655c3-kube-api-access-78p4d\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:43.492441 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.471341 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-workload-certs\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:43.492441 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.487945 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-envoy\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:43.492441 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.490863 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-podinfo\") pod \"e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:43.642209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.633128 5057 generic.go:356] "Generic (PLEG): container finished" podID="860644cd-397b-4af8-b425-a81491c9b66b" containerID="ab9c0e9c18b3005f83902c673219e92108a47242d93426f7163492bca609c233" exitCode=0 Oct 29 23:04:43.642209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.633295 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" event={"ID":"860644cd-397b-4af8-b425-a81491c9b66b","Type":"ContainerDied","Data":"ab9c0e9c18b3005f83902c673219e92108a47242d93426f7163492bca609c233"} Oct 29 23:04:43.642209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.633368 5057 scope.go:122] "RemoveContainer" containerID="0c141101a46c5d3ae43fd187abb8deabb90fa5e28d66d5dbc51f1096d107cd7f" Oct 29 23:04:43.708333 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.706007 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:43.985811 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.985784 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 23:04:43.999518 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.999479 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/860644cd-397b-4af8-b425-a81491c9b66b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860644cd-397b-4af8-b425-a81491c9b66b-istio-podinfo\") pod \"860644cd-397b-4af8-b425-a81491c9b66b\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " Oct 29 23:04:43.999697 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.999529 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860644cd-397b-4af8-b425-a81491c9b66b-kube-api-access-pg49f\" (UniqueName: \"kubernetes.io/projected/860644cd-397b-4af8-b425-a81491c9b66b-kube-api-access-pg49f\") pod \"860644cd-397b-4af8-b425-a81491c9b66b\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " Oct 29 23:04:43.999697 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.999618 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-istio-data\") pod \"860644cd-397b-4af8-b425-a81491c9b66b\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " Oct 29 23:04:43.999697 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.999673 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-istio-envoy\") pod \"860644cd-397b-4af8-b425-a81491c9b66b\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " Oct 29 23:04:43.999871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.999739 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/860644cd-397b-4af8-b425-a81491c9b66b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860644cd-397b-4af8-b425-a81491c9b66b-istiod-ca-cert\") pod \"860644cd-397b-4af8-b425-a81491c9b66b\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " Oct 29 23:04:43.999871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.999772 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-credential-socket\") pod \"860644cd-397b-4af8-b425-a81491c9b66b\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " Oct 29 23:04:43.999871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.999799 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-workload-certs\") pod \"860644cd-397b-4af8-b425-a81491c9b66b\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " Oct 29 23:04:44.000008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:43.999983 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-workload-socket\") pod \"860644cd-397b-4af8-b425-a81491c9b66b\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " Oct 29 23:04:44.000059 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.000044 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860644cd-397b-4af8-b425-a81491c9b66b-istio-token\" (UniqueName: \"kubernetes.io/projected/860644cd-397b-4af8-b425-a81491c9b66b-istio-token\") pod \"860644cd-397b-4af8-b425-a81491c9b66b\" (UID: \"860644cd-397b-4af8-b425-a81491c9b66b\") " Oct 29 23:04:44.002480 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.001602 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-istio-data" pod "860644cd-397b-4af8-b425-a81491c9b66b" (UID: "860644cd-397b-4af8-b425-a81491c9b66b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:44.002480 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.002023 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/860644cd-397b-4af8-b425-a81491c9b66b-istiod-ca-cert" pod "860644cd-397b-4af8-b425-a81491c9b66b" (UID: "860644cd-397b-4af8-b425-a81491c9b66b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:44.002480 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.002386 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-workload-socket" pod "860644cd-397b-4af8-b425-a81491c9b66b" (UID: "860644cd-397b-4af8-b425-a81491c9b66b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:44.002480 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.002453 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-workload-certs" pod "860644cd-397b-4af8-b425-a81491c9b66b" (UID: "860644cd-397b-4af8-b425-a81491c9b66b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:44.002886 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.002850 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-credential-socket" pod "860644cd-397b-4af8-b425-a81491c9b66b" (UID: "860644cd-397b-4af8-b425-a81491c9b66b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:44.004484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.004445 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860644cd-397b-4af8-b425-a81491c9b66b-kube-api-access-pg49f" pod "860644cd-397b-4af8-b425-a81491c9b66b" (UID: "860644cd-397b-4af8-b425-a81491c9b66b"). InnerVolumeSpecName "kube-api-access-pg49f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:44.005241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.005206 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860644cd-397b-4af8-b425-a81491c9b66b-istio-token" pod "860644cd-397b-4af8-b425-a81491c9b66b" (UID: "860644cd-397b-4af8-b425-a81491c9b66b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:44.006350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.006314 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/860644cd-397b-4af8-b425-a81491c9b66b-istio-podinfo" pod "860644cd-397b-4af8-b425-a81491c9b66b" (UID: "860644cd-397b-4af8-b425-a81491c9b66b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:44.007465 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.007437 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-istio-envoy" pod "860644cd-397b-4af8-b425-a81491c9b66b" (UID: "860644cd-397b-4af8-b425-a81491c9b66b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:44.104032 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.103934 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860644cd-397b-4af8-b425-a81491c9b66b-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:44.104335 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.104315 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg49f\" (UniqueName: \"kubernetes.io/projected/860644cd-397b-4af8-b425-a81491c9b66b-kube-api-access-pg49f\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:44.104482 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.104466 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:44.104572 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.104561 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:44.104664 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.104653 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860644cd-397b-4af8-b425-a81491c9b66b-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:44.104779 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.104764 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:44.105564 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.105542 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:44.105877 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.105686 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860644cd-397b-4af8-b425-a81491c9b66b-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:44.105877 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.105705 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860644cd-397b-4af8-b425-a81491c9b66b-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:44.113083 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:04:44.113040 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c26371_4cdf_4bf1_95d7_855d452655c3.slice/crio-bed97bc4b3b29056db0312884d6b3a9e20c4b27844aa598d890c4cbb9a64f7ce WatchSource:0}: Error finding container bed97bc4b3b29056db0312884d6b3a9e20c4b27844aa598d890c4cbb9a64f7ce: Status 404 returned error can't find the container with id bed97bc4b3b29056db0312884d6b3a9e20c4b27844aa598d890c4cbb9a64f7ce Oct 29 23:04:44.116450 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.116408 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:04:44.116533 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.116505 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:04:44.116533 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.116426 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t"] Oct 29 23:04:44.116603 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.116555 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:04:44.397466 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.396976 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 29 23:04:44.397466 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.397036 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" podUID="dcda00c4-c4da-43c2-b232-3f7c877c3ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 29 23:04:44.692094 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.692063 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" Oct 29 23:04:44.692094 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.692081 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb" event={"ID":"860644cd-397b-4af8-b425-a81491c9b66b","Type":"ContainerDied","Data":"bb55a90499dbdee488a69fedb3a886ee50131abcde80cdfb9c02582dba92e654"} Oct 29 23:04:44.692568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.692138 5057 scope.go:122] "RemoveContainer" containerID="ab9c0e9c18b3005f83902c673219e92108a47242d93426f7163492bca609c233" Oct 29 23:04:44.699717 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.699686 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" event={"ID":"06c26371-4cdf-4bf1-95d7-855d452655c3","Type":"ContainerStarted","Data":"95de34c0859f5b7e5cbf48288a72e2d852ca439d02b673d5b3619997dec7168e"} Oct 29 23:04:44.699843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.699741 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" event={"ID":"06c26371-4cdf-4bf1-95d7-855d452655c3","Type":"ContainerStarted","Data":"bed97bc4b3b29056db0312884d6b3a9e20c4b27844aa598d890c4cbb9a64f7ce"} Oct 29 23:04:44.708299 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.708278 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:44.767094 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.766875 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" podStartSLOduration=2.766855633 podStartE2EDuration="2.766855633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:44.765828214 +0000 UTC m=+1331.417326737" watchObservedRunningTime="2026-10-29 23:04:44.766855633 +0000 UTC m=+1331.418354164" Oct 29 23:04:44.812367 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.812340 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb"] Oct 29 23:04:44.819413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:44.819382 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-475876-openshift-default-8488756dfc-mfkgb"] Oct 29 23:04:45.397318 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.397284 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 29 23:04:45.397561 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.397531 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" podUID="dcda00c4-c4da-43c2-b232-3f7c877c3ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 29 23:04:45.418074 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.418045 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:04:45.495450 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.495280 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="860644cd-397b-4af8-b425-a81491c9b66b" path="/var/lib/kubelet/pods/860644cd-397b-4af8-b425-a81491c9b66b/volumes" Oct 29 23:04:45.502974 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.502942 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-workload-socket" pod "9864c587-073a-4b1e-9e7c-68fe058b3c16" (UID: "9864c587-073a-4b1e-9e7c-68fe058b3c16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:45.503339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.503313 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-workload-socket\") pod \"9864c587-073a-4b1e-9e7c-68fe058b3c16\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " Oct 29 23:04:45.508541 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.508506 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-token\" (UniqueName: \"kubernetes.io/projected/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-token\") pod \"9864c587-073a-4b1e-9e7c-68fe058b3c16\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " Oct 29 23:04:45.508967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.508952 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9864c587-073a-4b1e-9e7c-68fe058b3c16-kube-api-access-gq66n\" (UniqueName: \"kubernetes.io/projected/9864c587-073a-4b1e-9e7c-68fe058b3c16-kube-api-access-gq66n\") pod \"9864c587-073a-4b1e-9e7c-68fe058b3c16\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " Oct 29 23:04:45.510124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.510105 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-envoy\") pod \"9864c587-073a-4b1e-9e7c-68fe058b3c16\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " Oct 29 23:04:45.510439 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.510306 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-workload-certs\") pod \"9864c587-073a-4b1e-9e7c-68fe058b3c16\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " Oct 29 23:04:45.510439 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.510358 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9864c587-073a-4b1e-9e7c-68fe058b3c16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9864c587-073a-4b1e-9e7c-68fe058b3c16-istiod-ca-cert\") pod \"9864c587-073a-4b1e-9e7c-68fe058b3c16\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " Oct 29 23:04:45.510439 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.510387 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-credential-socket\") pod \"9864c587-073a-4b1e-9e7c-68fe058b3c16\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " Oct 29 23:04:45.510626 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.510441 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-podinfo\") pod \"9864c587-073a-4b1e-9e7c-68fe058b3c16\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " Oct 29 23:04:45.510679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.510626 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-workload-certs" pod "9864c587-073a-4b1e-9e7c-68fe058b3c16" (UID: "9864c587-073a-4b1e-9e7c-68fe058b3c16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:45.510679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.510648 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-data\") pod \"9864c587-073a-4b1e-9e7c-68fe058b3c16\" (UID: \"9864c587-073a-4b1e-9e7c-68fe058b3c16\") " Oct 29 23:04:45.510853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.510829 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-credential-socket" pod "9864c587-073a-4b1e-9e7c-68fe058b3c16" (UID: "9864c587-073a-4b1e-9e7c-68fe058b3c16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:45.511222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.511134 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9864c587-073a-4b1e-9e7c-68fe058b3c16-istiod-ca-cert" pod "9864c587-073a-4b1e-9e7c-68fe058b3c16" (UID: "9864c587-073a-4b1e-9e7c-68fe058b3c16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:45.512574 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.512551 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.512694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.512680 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.512797 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.512785 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9864c587-073a-4b1e-9e7c-68fe058b3c16-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.512895 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.512883 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.512976 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.512799 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-data" pod "9864c587-073a-4b1e-9e7c-68fe058b3c16" (UID: "9864c587-073a-4b1e-9e7c-68fe058b3c16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:45.519751 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.519653 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-token" pod "9864c587-073a-4b1e-9e7c-68fe058b3c16" (UID: "9864c587-073a-4b1e-9e7c-68fe058b3c16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:45.526352 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.526324 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-envoy" pod "9864c587-073a-4b1e-9e7c-68fe058b3c16" (UID: "9864c587-073a-4b1e-9e7c-68fe058b3c16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:45.526468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.526449 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9864c587-073a-4b1e-9e7c-68fe058b3c16-kube-api-access-gq66n" pod "9864c587-073a-4b1e-9e7c-68fe058b3c16" (UID: "9864c587-073a-4b1e-9e7c-68fe058b3c16"). InnerVolumeSpecName "kube-api-access-gq66n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:45.526895 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.526738 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-podinfo" pod "9864c587-073a-4b1e-9e7c-68fe058b3c16" (UID: "9864c587-073a-4b1e-9e7c-68fe058b3c16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:45.614901 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.614865 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.614901 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.614895 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.614901 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.614905 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq66n\" (UniqueName: \"kubernetes.io/projected/9864c587-073a-4b1e-9e7c-68fe058b3c16-kube-api-access-gq66n\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.615302 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.614915 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.615302 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.614923 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9864c587-073a-4b1e-9e7c-68fe058b3c16-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.705426 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.705395 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:04:45.708940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.708907 5057 generic.go:356] "Generic (PLEG): container finished" podID="bbfc0179-11dd-431d-af1a-0793c48ba6fe" containerID="cdc998dc73604bdd420660d1c09d263e9d4eb611a87e24c91785b95080835cbf" exitCode=0 Oct 29 23:04:45.709098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.708976 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" Oct 29 23:04:45.709098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.708986 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" event={"ID":"bbfc0179-11dd-431d-af1a-0793c48ba6fe","Type":"ContainerDied","Data":"cdc998dc73604bdd420660d1c09d263e9d4eb611a87e24c91785b95080835cbf"} Oct 29 23:04:45.709098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.709043 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc" event={"ID":"bbfc0179-11dd-431d-af1a-0793c48ba6fe","Type":"ContainerDied","Data":"74bdf25bab4334f36a532dc61d861968c2f75187525a3392a8a814aff46a80df"} Oct 29 23:04:45.709098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.709075 5057 scope.go:122] "RemoveContainer" containerID="cdc998dc73604bdd420660d1c09d263e9d4eb611a87e24c91785b95080835cbf" Oct 29 23:04:45.712345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.712318 5057 generic.go:356] "Generic (PLEG): container finished" podID="9864c587-073a-4b1e-9e7c-68fe058b3c16" containerID="d300568f26076396d8067fcd3f755a541b30555047ed82d19994391a1e43440a" exitCode=0 Oct 29 23:04:45.712459 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.712401 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" event={"ID":"9864c587-073a-4b1e-9e7c-68fe058b3c16","Type":"ContainerDied","Data":"d300568f26076396d8067fcd3f755a541b30555047ed82d19994391a1e43440a"} Oct 29 23:04:45.712459 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.712452 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" event={"ID":"9864c587-073a-4b1e-9e7c-68fe058b3c16","Type":"ContainerDied","Data":"208d919a9e21d16de982e7403d756f05b7ef22f653637e0cf903f879b304ad60"} Oct 29 23:04:45.712611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.712592 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd" Oct 29 23:04:45.714345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.714317 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:45.722864 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.722842 5057 scope.go:122] "RemoveContainer" containerID="cdc998dc73604bdd420660d1c09d263e9d4eb611a87e24c91785b95080835cbf" Oct 29 23:04:45.723434 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:04:45.723163 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdc998dc73604bdd420660d1c09d263e9d4eb611a87e24c91785b95080835cbf\": container with ID starting with cdc998dc73604bdd420660d1c09d263e9d4eb611a87e24c91785b95080835cbf not found: ID does not exist" containerID="cdc998dc73604bdd420660d1c09d263e9d4eb611a87e24c91785b95080835cbf" Oct 29 23:04:45.723434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.723226 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc998dc73604bdd420660d1c09d263e9d4eb611a87e24c91785b95080835cbf"} err="failed to get container status \"cdc998dc73604bdd420660d1c09d263e9d4eb611a87e24c91785b95080835cbf\": rpc error: code = NotFound desc = could not find container \"cdc998dc73604bdd420660d1c09d263e9d4eb611a87e24c91785b95080835cbf\": container with ID starting with cdc998dc73604bdd420660d1c09d263e9d4eb611a87e24c91785b95080835cbf not found: ID does not exist" Oct 29 23:04:45.723434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.723250 5057 scope.go:122] "RemoveContainer" containerID="d300568f26076396d8067fcd3f755a541b30555047ed82d19994391a1e43440a" Oct 29 23:04:45.734525 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.734505 5057 scope.go:122] "RemoveContainer" containerID="9603457875dc61f1e283129cf13a3ff129ec11b88905bc93ac83fe06f50fcf27" Oct 29 23:04:45.744252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.744230 5057 scope.go:122] "RemoveContainer" containerID="d300568f26076396d8067fcd3f755a541b30555047ed82d19994391a1e43440a" Oct 29 23:04:45.744688 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:04:45.744667 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d300568f26076396d8067fcd3f755a541b30555047ed82d19994391a1e43440a\": container with ID starting with d300568f26076396d8067fcd3f755a541b30555047ed82d19994391a1e43440a not found: ID does not exist" containerID="d300568f26076396d8067fcd3f755a541b30555047ed82d19994391a1e43440a" Oct 29 23:04:45.744778 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.744699 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d300568f26076396d8067fcd3f755a541b30555047ed82d19994391a1e43440a"} err="failed to get container status \"d300568f26076396d8067fcd3f755a541b30555047ed82d19994391a1e43440a\": rpc error: code = NotFound desc = could not find container \"d300568f26076396d8067fcd3f755a541b30555047ed82d19994391a1e43440a\": container with ID starting with d300568f26076396d8067fcd3f755a541b30555047ed82d19994391a1e43440a not found: ID does not exist" Oct 29 23:04:45.744778 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.744723 5057 scope.go:122] "RemoveContainer" containerID="9603457875dc61f1e283129cf13a3ff129ec11b88905bc93ac83fe06f50fcf27" Oct 29 23:04:45.745012 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:04:45.744993 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9603457875dc61f1e283129cf13a3ff129ec11b88905bc93ac83fe06f50fcf27\": container with ID starting with 9603457875dc61f1e283129cf13a3ff129ec11b88905bc93ac83fe06f50fcf27 not found: ID does not exist" containerID="9603457875dc61f1e283129cf13a3ff129ec11b88905bc93ac83fe06f50fcf27" Oct 29 23:04:45.745047 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.745019 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9603457875dc61f1e283129cf13a3ff129ec11b88905bc93ac83fe06f50fcf27"} err="failed to get container status \"9603457875dc61f1e283129cf13a3ff129ec11b88905bc93ac83fe06f50fcf27\": rpc error: code = NotFound desc = could not find container \"9603457875dc61f1e283129cf13a3ff129ec11b88905bc93ac83fe06f50fcf27\": container with ID starting with 9603457875dc61f1e283129cf13a3ff129ec11b88905bc93ac83fe06f50fcf27 not found: ID does not exist" Oct 29 23:04:45.974209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.948814 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-data\") pod \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " Oct 29 23:04:45.974567 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.974538 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-podinfo\") pod \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " Oct 29 23:04:45.974729 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.974713 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbfc0179-11dd-431d-af1a-0793c48ba6fe-kube-api-access-rsrfg\" (UniqueName: \"kubernetes.io/projected/bbfc0179-11dd-431d-af1a-0793c48ba6fe-kube-api-access-rsrfg\") pod \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " Oct 29 23:04:45.974880 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.974864 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-workload-socket\") pod \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " Oct 29 23:04:45.975066 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.975049 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istiod-ca-cert\") pod \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " Oct 29 23:04:45.975362 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.975346 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-workload-certs\") pod \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " Oct 29 23:04:45.991207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.975655 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-credential-socket\") pod \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " Oct 29 23:04:45.996356 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.996328 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-token\" (UniqueName: \"kubernetes.io/projected/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-token\") pod \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " Oct 29 23:04:45.996601 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.996582 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-envoy\") pod \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\" (UID: \"bbfc0179-11dd-431d-af1a-0793c48ba6fe\") " Oct 29 23:04:46.013878 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.977305 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-data" pod "bbfc0179-11dd-431d-af1a-0793c48ba6fe" (UID: "bbfc0179-11dd-431d-af1a-0793c48ba6fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:46.013878 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.989277 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-workload-socket" pod "bbfc0179-11dd-431d-af1a-0793c48ba6fe" (UID: "bbfc0179-11dd-431d-af1a-0793c48ba6fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:46.013878 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.991607 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-credential-socket" pod "bbfc0179-11dd-431d-af1a-0793c48ba6fe" (UID: "bbfc0179-11dd-431d-af1a-0793c48ba6fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:46.013878 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.991965 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istiod-ca-cert" pod "bbfc0179-11dd-431d-af1a-0793c48ba6fe" (UID: "bbfc0179-11dd-431d-af1a-0793c48ba6fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:46.013878 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:45.996263 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-workload-certs" pod "bbfc0179-11dd-431d-af1a-0793c48ba6fe" (UID: "bbfc0179-11dd-431d-af1a-0793c48ba6fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:46.013878 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.013464 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-podinfo" pod "bbfc0179-11dd-431d-af1a-0793c48ba6fe" (UID: "bbfc0179-11dd-431d-af1a-0793c48ba6fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:46.014386 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.014299 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:46.014386 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.014329 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:46.014386 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.014342 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:46.014386 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.014354 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:46.014386 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.014366 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:46.014386 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.014382 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:46.017206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.017156 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbfc0179-11dd-431d-af1a-0793c48ba6fe-kube-api-access-rsrfg" pod "bbfc0179-11dd-431d-af1a-0793c48ba6fe" (UID: "bbfc0179-11dd-431d-af1a-0793c48ba6fe"). InnerVolumeSpecName "kube-api-access-rsrfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:46.023336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.023143 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-token" pod "bbfc0179-11dd-431d-af1a-0793c48ba6fe" (UID: "bbfc0179-11dd-431d-af1a-0793c48ba6fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:46.094445 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.084429 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-envoy" pod "bbfc0179-11dd-431d-af1a-0793c48ba6fe" (UID: "bbfc0179-11dd-431d-af1a-0793c48ba6fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:46.171259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.151833 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd"] Oct 29 23:04:46.213251 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.212849 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsrfg\" (UniqueName: \"kubernetes.io/projected/bbfc0179-11dd-431d-af1a-0793c48ba6fe-kube-api-access-rsrfg\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:46.213587 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.213566 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:46.213824 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.213810 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbfc0179-11dd-431d-af1a-0793c48ba6fe-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:46.233091 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.233059 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-t7qjd"] Oct 29 23:04:46.244881 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.244850 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:04:46.479958 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.466564 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:46.479958 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.475571 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:46.479958 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.479431 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:04:46.561993 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.561953 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc"] Oct 29 23:04:46.569931 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.569898 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-nsnzc"] Oct 29 23:04:46.763999 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.763895 5057 generic.go:356] "Generic (PLEG): container finished" podID="7cf5e014-85ba-4024-9ebc-088735098c5c" containerID="d53ce11747772ebcbc2daaefe98ce68370a0c18afac031ad86a492a97d69dee9" exitCode=0 Oct 29 23:04:46.764498 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.764086 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" event={"ID":"7cf5e014-85ba-4024-9ebc-088735098c5c","Type":"ContainerDied","Data":"d53ce11747772ebcbc2daaefe98ce68370a0c18afac031ad86a492a97d69dee9"} Oct 29 23:04:46.764498 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.764145 5057 scope.go:122] "RemoveContainer" containerID="834443d9a86fb7261754ef952239745c80d8ff100b89cffcbebc26741059c880" Oct 29 23:04:46.773297 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.773271 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:46.777172 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.776883 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:04:46.913971 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.913941 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:04:46.957235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.957149 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m"] Oct 29 23:04:46.958735 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.958705 5057 cpu_manager.go:414] "containerMap: removing container" podUID="860644cd-397b-4af8-b425-a81491c9b66b" containerName="istio-proxy" Oct 29 23:04:46.958735 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.958734 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860644cd-397b-4af8-b425-a81491c9b66b" containerName="istio-proxy" Oct 29 23:04:46.958951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.958765 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9864c587-073a-4b1e-9e7c-68fe058b3c16" containerName="istio-proxy" Oct 29 23:04:46.958951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.958774 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9864c587-073a-4b1e-9e7c-68fe058b3c16" containerName="istio-proxy" Oct 29 23:04:46.958951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.958798 5057 cpu_manager.go:414] "containerMap: removing container" podUID="860644cd-397b-4af8-b425-a81491c9b66b" containerName="istio-proxy" Oct 29 23:04:46.958951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.958806 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860644cd-397b-4af8-b425-a81491c9b66b" containerName="istio-proxy" Oct 29 23:04:46.958951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.958819 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7cf5e014-85ba-4024-9ebc-088735098c5c" containerName="istio-proxy" Oct 29 23:04:46.958951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.958827 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cf5e014-85ba-4024-9ebc-088735098c5c" containerName="istio-proxy" Oct 29 23:04:46.958951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.958840 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7cf5e014-85ba-4024-9ebc-088735098c5c" containerName="istio-proxy" Oct 29 23:04:46.958951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.958847 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cf5e014-85ba-4024-9ebc-088735098c5c" containerName="istio-proxy" Oct 29 23:04:46.958951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.958868 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9864c587-073a-4b1e-9e7c-68fe058b3c16" containerName="istio-proxy" Oct 29 23:04:46.958951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.958875 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9864c587-073a-4b1e-9e7c-68fe058b3c16" containerName="istio-proxy" Oct 29 23:04:46.958951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.958949 5057 cpu_manager.go:414] "containerMap: removing container" podUID="bbfc0179-11dd-431d-af1a-0793c48ba6fe" containerName="istio-proxy" Oct 29 23:04:46.958951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.958957 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbfc0179-11dd-431d-af1a-0793c48ba6fe" containerName="istio-proxy" Oct 29 23:04:46.959586 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.959449 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbfc0179-11dd-431d-af1a-0793c48ba6fe" containerName="istio-proxy" Oct 29 23:04:46.959586 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.959473 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="860644cd-397b-4af8-b425-a81491c9b66b" containerName="istio-proxy" Oct 29 23:04:46.959586 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.959484 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9864c587-073a-4b1e-9e7c-68fe058b3c16" containerName="istio-proxy" Oct 29 23:04:46.959586 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.959497 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cf5e014-85ba-4024-9ebc-088735098c5c" containerName="istio-proxy" Oct 29 23:04:46.959586 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.959508 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cf5e014-85ba-4024-9ebc-088735098c5c" containerName="istio-proxy" Oct 29 23:04:46.959586 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.959517 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9864c587-073a-4b1e-9e7c-68fe058b3c16" containerName="istio-proxy" Oct 29 23:04:46.982016 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.974876 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" Oct 29 23:04:46.998374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.998072 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-41b12c-serving-cert\"" Oct 29 23:04:47.003950 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:46.998977 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m"] Oct 29 23:04:47.056064 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.055982 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-workload-certs\") pod \"7cf5e014-85ba-4024-9ebc-088735098c5c\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " Oct 29 23:04:47.056235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.056094 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-data\") pod \"7cf5e014-85ba-4024-9ebc-088735098c5c\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " Oct 29 23:04:47.056235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.056135 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-podinfo\") pod \"7cf5e014-85ba-4024-9ebc-088735098c5c\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " Oct 29 23:04:47.056373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.056236 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-credential-socket\") pod \"7cf5e014-85ba-4024-9ebc-088735098c5c\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " Oct 29 23:04:47.056373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.056314 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-workload-certs" pod "7cf5e014-85ba-4024-9ebc-088735098c5c" (UID: "7cf5e014-85ba-4024-9ebc-088735098c5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:47.056373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.056326 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-envoy\") pod \"7cf5e014-85ba-4024-9ebc-088735098c5c\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " Oct 29 23:04:47.056525 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.056428 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-workload-socket\") pod \"7cf5e014-85ba-4024-9ebc-088735098c5c\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " Oct 29 23:04:47.056525 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.056461 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-data" pod "7cf5e014-85ba-4024-9ebc-088735098c5c" (UID: "7cf5e014-85ba-4024-9ebc-088735098c5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:47.056525 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.056509 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cf5e014-85ba-4024-9ebc-088735098c5c-kube-api-access-sqzmp\" (UniqueName: \"kubernetes.io/projected/7cf5e014-85ba-4024-9ebc-088735098c5c-kube-api-access-sqzmp\") pod \"7cf5e014-85ba-4024-9ebc-088735098c5c\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " Oct 29 23:04:47.056672 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.056569 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-token\" (UniqueName: \"kubernetes.io/projected/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-token\") pod \"7cf5e014-85ba-4024-9ebc-088735098c5c\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " Oct 29 23:04:47.056672 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.056647 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cf5e014-85ba-4024-9ebc-088735098c5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cf5e014-85ba-4024-9ebc-088735098c5c-istiod-ca-cert\") pod \"7cf5e014-85ba-4024-9ebc-088735098c5c\" (UID: \"7cf5e014-85ba-4024-9ebc-088735098c5c\") " Oct 29 23:04:47.056766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.056719 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-workload-socket" pod "7cf5e014-85ba-4024-9ebc-088735098c5c" (UID: "7cf5e014-85ba-4024-9ebc-088735098c5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:47.056831 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.056806 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-credential-socket" pod "7cf5e014-85ba-4024-9ebc-088735098c5c" (UID: "7cf5e014-85ba-4024-9ebc-088735098c5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:47.057163 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.057126 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90cef309-7998-4ca4-adff-acd2e5d5ffc2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m\" (UID: \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" Oct 29 23:04:47.057296 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.057157 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cf5e014-85ba-4024-9ebc-088735098c5c-istiod-ca-cert" pod "7cf5e014-85ba-4024-9ebc-088735098c5c" (UID: "7cf5e014-85ba-4024-9ebc-088735098c5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:47.057425 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.057405 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk7hh\" (UniqueName: \"kubernetes.io/projected/90cef309-7998-4ca4-adff-acd2e5d5ffc2-kube-api-access-sk7hh\") pod \"maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m\" (UID: \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" Oct 29 23:04:47.057530 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.057515 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cef309-7998-4ca4-adff-acd2e5d5ffc2-metrics-certs\") pod \"maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m\" (UID: \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" Oct 29 23:04:47.057772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.057752 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90cef309-7998-4ca4-adff-acd2e5d5ffc2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m\" (UID: \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" Oct 29 23:04:47.058504 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.058483 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:47.058644 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.058629 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:47.058737 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.058725 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cf5e014-85ba-4024-9ebc-088735098c5c-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:47.058945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.058931 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:47.059053 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.059042 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:47.059307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.059267 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-podinfo" pod "7cf5e014-85ba-4024-9ebc-088735098c5c" (UID: "7cf5e014-85ba-4024-9ebc-088735098c5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:47.059842 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.059820 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-token" pod "7cf5e014-85ba-4024-9ebc-088735098c5c" (UID: "7cf5e014-85ba-4024-9ebc-088735098c5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:47.060006 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.059989 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-envoy" pod "7cf5e014-85ba-4024-9ebc-088735098c5c" (UID: "7cf5e014-85ba-4024-9ebc-088735098c5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:47.062249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.062224 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf5e014-85ba-4024-9ebc-088735098c5c-kube-api-access-sqzmp" pod "7cf5e014-85ba-4024-9ebc-088735098c5c" (UID: "7cf5e014-85ba-4024-9ebc-088735098c5c"). InnerVolumeSpecName "kube-api-access-sqzmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:47.172779 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.170407 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90cef309-7998-4ca4-adff-acd2e5d5ffc2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m\" (UID: \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" Oct 29 23:04:47.172779 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.170640 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk7hh\" (UniqueName: \"kubernetes.io/projected/90cef309-7998-4ca4-adff-acd2e5d5ffc2-kube-api-access-sk7hh\") pod \"maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m\" (UID: \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" Oct 29 23:04:47.172779 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.170740 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cef309-7998-4ca4-adff-acd2e5d5ffc2-metrics-certs\") pod \"maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m\" (UID: \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" Oct 29 23:04:47.172779 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.170857 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90cef309-7998-4ca4-adff-acd2e5d5ffc2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m\" (UID: \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" Oct 29 23:04:47.172779 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.171001 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:47.172779 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.171021 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqzmp\" (UniqueName: \"kubernetes.io/projected/7cf5e014-85ba-4024-9ebc-088735098c5c-kube-api-access-sqzmp\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:47.172779 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.171036 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:47.172779 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.171050 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cf5e014-85ba-4024-9ebc-088735098c5c-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:47.172779 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:04:47.171160 5057 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-serving-cert: secret "maas-api-e2e-ipp-b-41b12c-serving-cert" not found Oct 29 23:04:47.198214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.174343 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90cef309-7998-4ca4-adff-acd2e5d5ffc2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m\" (UID: \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" Oct 29 23:04:47.198214 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:04:47.178262 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90cef309-7998-4ca4-adff-acd2e5d5ffc2-maas-api-tls podName:90cef309-7998-4ca4-adff-acd2e5d5ffc2 nodeName:}" failed. No retries permitted until 2026-10-29 23:04:47.671221052 +0000 UTC m=+1334.322719566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90cef309-7998-4ca4-adff-acd2e5d5ffc2-maas-api-tls") pod "maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" (UID: "90cef309-7998-4ca4-adff-acd2e5d5ffc2") : secret "maas-api-e2e-ipp-b-41b12c-serving-cert" not found Oct 29 23:04:47.198214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.183213 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cef309-7998-4ca4-adff-acd2e5d5ffc2-metrics-certs\") pod \"maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m\" (UID: \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" Oct 29 23:04:47.239742 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.231527 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk7hh\" (UniqueName: \"kubernetes.io/projected/90cef309-7998-4ca4-adff-acd2e5d5ffc2-kube-api-access-sk7hh\") pod \"maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m\" (UID: \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" Oct 29 23:04:47.484388 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.484353 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9864c587-073a-4b1e-9e7c-68fe058b3c16" path="/var/lib/kubelet/pods/9864c587-073a-4b1e-9e7c-68fe058b3c16/volumes" Oct 29 23:04:47.484861 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.484847 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbfc0179-11dd-431d-af1a-0793c48ba6fe" path="/var/lib/kubelet/pods/bbfc0179-11dd-431d-af1a-0793c48ba6fe/volumes" Oct 29 23:04:47.685998 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.685954 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90cef309-7998-4ca4-adff-acd2e5d5ffc2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m\" (UID: \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" Oct 29 23:04:47.745264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.711640 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90cef309-7998-4ca4-adff-acd2e5d5ffc2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m\" (UID: \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" Oct 29 23:04:47.821218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.807735 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" Oct 29 23:04:47.822889 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.813256 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq" event={"ID":"7cf5e014-85ba-4024-9ebc-088735098c5c","Type":"ContainerDied","Data":"10ebdf4740e472cbca5c1e9406dd54176a12be46aa11cb1bc67dd5585538f0fc"} Oct 29 23:04:47.823111 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:47.823095 5057 scope.go:122] "RemoveContainer" containerID="d53ce11747772ebcbc2daaefe98ce68370a0c18afac031ad86a492a97d69dee9" Oct 29 23:04:48.005971 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:48.005881 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" Oct 29 23:04:48.078295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:48.075294 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq"] Oct 29 23:04:48.127498 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:48.127445 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-50734e-openshift-default-56977f9cd9-4pssq"] Oct 29 23:04:48.898261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:48.897848 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m"] Oct 29 23:04:49.390521 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:49.390483 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8"] Oct 29 23:04:49.393121 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:49.392712 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="860644cd-397b-4af8-b425-a81491c9b66b" containerName="istio-proxy" Oct 29 23:04:49.401446 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:49.401421 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8" Oct 29 23:04:49.421975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:49.421944 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-0941c8-dockercfg-txkqp\"" Oct 29 23:04:49.423998 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:49.423974 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-0941c8\"" Oct 29 23:04:49.478211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:49.469746 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8"] Oct 29 23:04:49.526210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:49.517645 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tnxd\" (UniqueName: \"kubernetes.io/projected/41347d77-4ec9-4e28-8651-6868480061c4-kube-api-access-9tnxd\") pod \"payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8\" (UID: \"41347d77-4ec9-4e28-8651-6868480061c4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8" Oct 29 23:04:49.526210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:49.517730 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41347d77-4ec9-4e28-8651-6868480061c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8\" (UID: \"41347d77-4ec9-4e28-8651-6868480061c4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8" Oct 29 23:04:49.692752 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:49.674826 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tnxd\" (UniqueName: \"kubernetes.io/projected/41347d77-4ec9-4e28-8651-6868480061c4-kube-api-access-9tnxd\") pod \"payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8\" (UID: \"41347d77-4ec9-4e28-8651-6868480061c4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8" Oct 29 23:04:49.692752 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:49.674902 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41347d77-4ec9-4e28-8651-6868480061c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8\" (UID: \"41347d77-4ec9-4e28-8651-6868480061c4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8" Oct 29 23:04:49.718292 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:49.693935 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41347d77-4ec9-4e28-8651-6868480061c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8\" (UID: \"41347d77-4ec9-4e28-8651-6868480061c4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8" Oct 29 23:04:50.140298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:50.135793 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cf5e014-85ba-4024-9ebc-088735098c5c" path="/var/lib/kubelet/pods/7cf5e014-85ba-4024-9ebc-088735098c5c/volumes" Oct 29 23:04:50.157266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:50.141707 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc"] Oct 29 23:04:50.252215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:50.231967 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tnxd\" (UniqueName: \"kubernetes.io/projected/41347d77-4ec9-4e28-8651-6868480061c4-kube-api-access-9tnxd\") pod \"payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8\" (UID: \"41347d77-4ec9-4e28-8651-6868480061c4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8" Oct 29 23:04:50.252215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:50.250655 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc"] Oct 29 23:04:50.252215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:50.250785 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc" Oct 29 23:04:50.306073 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:50.302814 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" event={"ID":"90cef309-7998-4ca4-adff-acd2e5d5ffc2","Type":"ContainerStarted","Data":"199026090e2480476f74d0cfecaadbaa4b47c28f095e125402919baf01015bce"} Oct 29 23:04:50.385498 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:50.349843 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8" Oct 29 23:04:50.433269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:50.395096 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2v6f\" (UniqueName: \"kubernetes.io/projected/5fa0b407-b14b-47c2-a6d7-a16536017857-kube-api-access-m2v6f\") pod \"payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc\" (UID: \"5fa0b407-b14b-47c2-a6d7-a16536017857\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc" Oct 29 23:04:50.433660 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:50.433608 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fa0b407-b14b-47c2-a6d7-a16536017857-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc\" (UID: \"5fa0b407-b14b-47c2-a6d7-a16536017857\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc" Oct 29 23:04:50.580218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:50.543996 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2v6f\" (UniqueName: \"kubernetes.io/projected/5fa0b407-b14b-47c2-a6d7-a16536017857-kube-api-access-m2v6f\") pod \"payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc\" (UID: \"5fa0b407-b14b-47c2-a6d7-a16536017857\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc" Oct 29 23:04:50.760179 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:50.760091 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fa0b407-b14b-47c2-a6d7-a16536017857-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc\" (UID: \"5fa0b407-b14b-47c2-a6d7-a16536017857\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc" Oct 29 23:04:50.796172 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:50.761348 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fa0b407-b14b-47c2-a6d7-a16536017857-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc\" (UID: \"5fa0b407-b14b-47c2-a6d7-a16536017857\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc" Oct 29 23:04:50.874210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:50.867996 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2v6f\" (UniqueName: \"kubernetes.io/projected/5fa0b407-b14b-47c2-a6d7-a16536017857-kube-api-access-m2v6f\") pod \"payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc\" (UID: \"5fa0b407-b14b-47c2-a6d7-a16536017857\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc" Oct 29 23:04:50.988275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:50.929769 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc" Oct 29 23:04:51.502210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:51.501275 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs"] Oct 29 23:04:51.662057 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:51.656472 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:51.854843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:51.840248 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-workload-certs\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:51.854843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:51.840321 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-token\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:51.854843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:51.840428 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-envoy\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:51.854843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:51.840528 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-podinfo\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:51.854843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:51.840759 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-data\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:51.854843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:51.840909 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-credential-socket\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:51.854843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:51.840986 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istiod-ca-cert\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:51.854843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:51.841044 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxw9l\" (UniqueName: \"kubernetes.io/projected/34c31c14-402a-4cff-a9e1-0a6940ffc05a-kube-api-access-cxw9l\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:51.854843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:51.841089 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-workload-socket\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:51.921266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:51.920682 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:04:51.936918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:51.935748 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:04:51.959654 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:51.959610 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istiod-ca-cert\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:51.960053 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:51.960026 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxw9l\" (UniqueName: \"kubernetes.io/projected/34c31c14-402a-4cff-a9e1-0a6940ffc05a-kube-api-access-cxw9l\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:52.196990 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.182850 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-workload-socket\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:52.208877 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.201233 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-workload-certs\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:52.208877 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.201320 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-token\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:52.208877 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.201476 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-envoy\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:52.208877 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.201556 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-podinfo\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:52.208877 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.201639 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-data\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:52.208877 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.201767 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-credential-socket\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:52.208877 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.153105 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:04:52.210392 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.209766 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-workload-certs\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:52.232907 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.232522 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-credential-socket\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:52.324211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.310086 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-data\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:52.324211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:51.968907 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istiod-ca-cert\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:52.324840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.324631 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-workload-socket\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:52.455057 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.454939 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-618d6c-openshift-default-dockercfg-zdpwm\"" Oct 29 23:04:52.788975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.765736 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-envoy\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:52.795318 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.791234 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxw9l\" (UniqueName: \"kubernetes.io/projected/34c31c14-402a-4cff-a9e1-0a6940ffc05a-kube-api-access-cxw9l\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:52.796740 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.796674 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-podinfo\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:52.798302 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:04:52.798265 5057 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.143s" Oct 29 23:04:52.798593 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.798573 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs"] Oct 29 23:04:52.834449 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.834406 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx"] Oct 29 23:04:52.847244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.841988 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-token\") pod \"e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:52.940431 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.940317 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 23:04:52.944290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.941066 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:04:52.944290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.943307 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:04:52.944290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.943372 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:04:52.963047 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:52.962518 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl"] Oct 29 23:04:53.097106 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:53.096727 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:53.172219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:53.162461 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:04:53.381084 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:53.380959 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:04:53.404691 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:53.404669 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:04:53.508216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:53.485665 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" Oct 29 23:04:54.093223 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:04:54.082396 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa0b407_b14b_47c2_a6d7_a16536017857.slice/crio-54d9f49641fed6a1dd609d8fc9dd40f566949795c86d4d33fc73969ad7a73d60 WatchSource:0}: Error finding container 54d9f49641fed6a1dd609d8fc9dd40f566949795c86d4d33fc73969ad7a73d60: Status 404 returned error can't find the container with id 54d9f49641fed6a1dd609d8fc9dd40f566949795c86d4d33fc73969ad7a73d60 Oct 29 23:04:54.119881 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:54.119823 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" podStartSLOduration=7.024225265 podStartE2EDuration="8.119803126s" totalImagesPullingTime="1.095577861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:53.718742588 +0000 UTC m=+1340.370241123" watchObservedRunningTime="2026-10-29 23:04:54.119803126 +0000 UTC m=+1340.771301656" Oct 29 23:04:54.207068 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:54.207046 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc"] Oct 29 23:04:54.209221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:54.209178 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8"] Oct 29 23:04:54.506558 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:54.505544 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs"] Oct 29 23:04:54.520052 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:54.520009 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:04:54.520146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:54.520128 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:04:54.520214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:54.520172 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:04:54.603011 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:54.602975 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8" event={"ID":"41347d77-4ec9-4e28-8651-6868480061c4","Type":"ContainerStarted","Data":"2ec5962684d0c3ce2a5ec684dce6adbef4fde9b9bb20d52e80761db847674abf"} Oct 29 23:04:54.610481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:54.610451 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" event={"ID":"90cef309-7998-4ca4-adff-acd2e5d5ffc2","Type":"ContainerStarted","Data":"f4208c5e34c71fc10bfc65d0aa810f8878a80973d437be915e0a492bd02be79c"} Oct 29 23:04:54.613596 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:54.613572 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc" event={"ID":"5fa0b407-b14b-47c2-a6d7-a16536017857","Type":"ContainerStarted","Data":"54d9f49641fed6a1dd609d8fc9dd40f566949795c86d4d33fc73969ad7a73d60"} Oct 29 23:04:54.614144 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:54.614118 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc" Oct 29 23:04:54.699969 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:54.699869 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc" podStartSLOduration=5.699794547 podStartE2EDuration="5.699794547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:54.678785269 +0000 UTC m=+1341.330283800" watchObservedRunningTime="2026-10-29 23:04:54.699794547 +0000 UTC m=+1341.351293078" Oct 29 23:04:55.682209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:55.681851 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" event={"ID":"34c31c14-402a-4cff-a9e1-0a6940ffc05a","Type":"ContainerStarted","Data":"b781f2aa745b0adda57e926bfbcc188a00e31148dfd3d50dba58ab4db389f26b"} Oct 29 23:04:55.682209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:55.681924 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" event={"ID":"34c31c14-402a-4cff-a9e1-0a6940ffc05a","Type":"ContainerStarted","Data":"6690c478c526fef225607f7a442c096b98c6071e9b354b7517e1914483ecb265"} Oct 29 23:04:55.685526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:55.685492 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8" event={"ID":"41347d77-4ec9-4e28-8651-6868480061c4","Type":"ContainerStarted","Data":"1d4c57ca2fb3f6789f3964c8b38d46e58225853e87355ff3575ddb7da50e8885"} Oct 29 23:04:55.693416 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:55.686586 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8" Oct 29 23:04:55.709689 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:55.709592 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc" event={"ID":"5fa0b407-b14b-47c2-a6d7-a16536017857","Type":"ContainerStarted","Data":"3ef171e02c2b37132106337f7542134a572ad231a7c510ee817d0a40579e69a2"} Oct 29 23:04:55.752208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:55.751858 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" podStartSLOduration=4.751838394 podStartE2EDuration="4.751838394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:55.749361352 +0000 UTC m=+1342.400859893" watchObservedRunningTime="2026-10-29 23:04:55.751838394 +0000 UTC m=+1342.403336925" Oct 29 23:04:55.896949 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:55.894581 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8" podStartSLOduration=6.894559495 podStartE2EDuration="6.894559495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:55.825610065 +0000 UTC m=+1342.477108596" watchObservedRunningTime="2026-10-29 23:04:55.894559495 +0000 UTC m=+1342.546058029" Oct 29 23:04:56.109280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:56.104438 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:56.121528 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:56.121496 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:56.720781 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:56.719837 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:56.721332 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:56.720858 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:04:57.841214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:57.840516 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx" podUID="79d001db-e9ee-46aa-9475-5306f125d606" containerName="payload-processing" containerID="cri-o://fffee417c15d544fcec34526b583c155c2bcf8c585f74772b8f5de9c99eae87a" gracePeriod=25 Oct 29 23:04:57.976253 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:57.963355 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl" podUID="16e0999a-213d-4f93-bbd4-3c007ae1d7eb" containerName="payload-pre-processing" containerID="cri-o://b6fe955af1f3c1e1ad4b3b8f8246fec2e82294a936ffe75f0c529191a267b446" gracePeriod=25 Oct 29 23:04:58.572491 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.568453 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx" Oct 29 23:04:58.606588 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.606126 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d001db-e9ee-46aa-9475-5306f125d606-kube-api-access-8sfrw\" (UniqueName: \"kubernetes.io/projected/79d001db-e9ee-46aa-9475-5306f125d606-kube-api-access-8sfrw\") pod \"79d001db-e9ee-46aa-9475-5306f125d606\" (UID: \"79d001db-e9ee-46aa-9475-5306f125d606\") " Oct 29 23:04:58.606588 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.606198 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d001db-e9ee-46aa-9475-5306f125d606-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d001db-e9ee-46aa-9475-5306f125d606-plugins-config-volume\") pod \"79d001db-e9ee-46aa-9475-5306f125d606\" (UID: \"79d001db-e9ee-46aa-9475-5306f125d606\") " Oct 29 23:04:58.607245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.607090 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d001db-e9ee-46aa-9475-5306f125d606-plugins-config-volume" pod "79d001db-e9ee-46aa-9475-5306f125d606" (UID: "79d001db-e9ee-46aa-9475-5306f125d606"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:58.630832 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.630779 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d001db-e9ee-46aa-9475-5306f125d606-kube-api-access-8sfrw" pod "79d001db-e9ee-46aa-9475-5306f125d606" (UID: "79d001db-e9ee-46aa-9475-5306f125d606"). InnerVolumeSpecName "kube-api-access-8sfrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:58.707638 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.707594 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl" Oct 29 23:04:58.709337 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.709310 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sfrw\" (UniqueName: \"kubernetes.io/projected/79d001db-e9ee-46aa-9475-5306f125d606-kube-api-access-8sfrw\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:58.709440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.709342 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d001db-e9ee-46aa-9475-5306f125d606-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:58.810262 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.810211 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e0999a-213d-4f93-bbd4-3c007ae1d7eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e0999a-213d-4f93-bbd4-3c007ae1d7eb-plugins-config-volume\") pod \"16e0999a-213d-4f93-bbd4-3c007ae1d7eb\" (UID: \"16e0999a-213d-4f93-bbd4-3c007ae1d7eb\") " Oct 29 23:04:58.810447 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.810328 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e0999a-213d-4f93-bbd4-3c007ae1d7eb-kube-api-access-5qjfg\" (UniqueName: \"kubernetes.io/projected/16e0999a-213d-4f93-bbd4-3c007ae1d7eb-kube-api-access-5qjfg\") pod \"16e0999a-213d-4f93-bbd4-3c007ae1d7eb\" (UID: \"16e0999a-213d-4f93-bbd4-3c007ae1d7eb\") " Oct 29 23:04:58.810673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.810642 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e0999a-213d-4f93-bbd4-3c007ae1d7eb-plugins-config-volume" pod "16e0999a-213d-4f93-bbd4-3c007ae1d7eb" (UID: "16e0999a-213d-4f93-bbd4-3c007ae1d7eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:58.811481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.811454 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e0999a-213d-4f93-bbd4-3c007ae1d7eb-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:58.812709 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.812679 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e0999a-213d-4f93-bbd4-3c007ae1d7eb-kube-api-access-5qjfg" pod "16e0999a-213d-4f93-bbd4-3c007ae1d7eb" (UID: "16e0999a-213d-4f93-bbd4-3c007ae1d7eb"). InnerVolumeSpecName "kube-api-access-5qjfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:58.913562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.913521 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qjfg\" (UniqueName: \"kubernetes.io/projected/16e0999a-213d-4f93-bbd4-3c007ae1d7eb-kube-api-access-5qjfg\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:04:58.995056 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.995016 5057 generic.go:356] "Generic (PLEG): container finished" podID="16e0999a-213d-4f93-bbd4-3c007ae1d7eb" containerID="b6fe955af1f3c1e1ad4b3b8f8246fec2e82294a936ffe75f0c529191a267b446" exitCode=0 Oct 29 23:04:58.995255 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.995127 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl" Oct 29 23:04:58.995337 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.995277 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl" event={"ID":"16e0999a-213d-4f93-bbd4-3c007ae1d7eb","Type":"ContainerDied","Data":"b6fe955af1f3c1e1ad4b3b8f8246fec2e82294a936ffe75f0c529191a267b446"} Oct 29 23:04:58.995397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.995379 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl" event={"ID":"16e0999a-213d-4f93-bbd4-3c007ae1d7eb","Type":"ContainerDied","Data":"3811878037b1d6a4cbb8854d0be422b7238080ff6bc0a456e771e5d67bfc80ee"} Oct 29 23:04:58.995534 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.995422 5057 scope.go:122] "RemoveContainer" containerID="b6fe955af1f3c1e1ad4b3b8f8246fec2e82294a936ffe75f0c529191a267b446" Oct 29 23:04:58.998022 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.997995 5057 generic.go:356] "Generic (PLEG): container finished" podID="79d001db-e9ee-46aa-9475-5306f125d606" containerID="fffee417c15d544fcec34526b583c155c2bcf8c585f74772b8f5de9c99eae87a" exitCode=0 Oct 29 23:04:58.999655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.999634 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx" Oct 29 23:04:59.000818 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:58.999666 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx" event={"ID":"79d001db-e9ee-46aa-9475-5306f125d606","Type":"ContainerDied","Data":"fffee417c15d544fcec34526b583c155c2bcf8c585f74772b8f5de9c99eae87a"} Oct 29 23:04:59.000991 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:59.000971 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx" event={"ID":"79d001db-e9ee-46aa-9475-5306f125d606","Type":"ContainerDied","Data":"26d8a8f40d32ee4fecb9f15db1540f51e2c6513904ca345b669d169092599e42"} Oct 29 23:04:59.014128 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:59.014102 5057 scope.go:122] "RemoveContainer" containerID="b6fe955af1f3c1e1ad4b3b8f8246fec2e82294a936ffe75f0c529191a267b446" Oct 29 23:04:59.016343 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:04:59.016317 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6fe955af1f3c1e1ad4b3b8f8246fec2e82294a936ffe75f0c529191a267b446\": container with ID starting with b6fe955af1f3c1e1ad4b3b8f8246fec2e82294a936ffe75f0c529191a267b446 not found: ID does not exist" containerID="b6fe955af1f3c1e1ad4b3b8f8246fec2e82294a936ffe75f0c529191a267b446" Oct 29 23:04:59.016449 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:59.016351 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6fe955af1f3c1e1ad4b3b8f8246fec2e82294a936ffe75f0c529191a267b446"} err="failed to get container status \"b6fe955af1f3c1e1ad4b3b8f8246fec2e82294a936ffe75f0c529191a267b446\": rpc error: code = NotFound desc = could not find container \"b6fe955af1f3c1e1ad4b3b8f8246fec2e82294a936ffe75f0c529191a267b446\": container with ID starting with b6fe955af1f3c1e1ad4b3b8f8246fec2e82294a936ffe75f0c529191a267b446 not found: ID does not exist" Oct 29 23:04:59.016449 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:59.016369 5057 scope.go:122] "RemoveContainer" containerID="fffee417c15d544fcec34526b583c155c2bcf8c585f74772b8f5de9c99eae87a" Oct 29 23:04:59.029524 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:59.029498 5057 scope.go:122] "RemoveContainer" containerID="fffee417c15d544fcec34526b583c155c2bcf8c585f74772b8f5de9c99eae87a" Oct 29 23:04:59.030149 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:04:59.030021 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fffee417c15d544fcec34526b583c155c2bcf8c585f74772b8f5de9c99eae87a\": container with ID starting with fffee417c15d544fcec34526b583c155c2bcf8c585f74772b8f5de9c99eae87a not found: ID does not exist" containerID="fffee417c15d544fcec34526b583c155c2bcf8c585f74772b8f5de9c99eae87a" Oct 29 23:04:59.030149 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:59.030071 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fffee417c15d544fcec34526b583c155c2bcf8c585f74772b8f5de9c99eae87a"} err="failed to get container status \"fffee417c15d544fcec34526b583c155c2bcf8c585f74772b8f5de9c99eae87a\": rpc error: code = NotFound desc = could not find container \"fffee417c15d544fcec34526b583c155c2bcf8c585f74772b8f5de9c99eae87a\": container with ID starting with fffee417c15d544fcec34526b583c155c2bcf8c585f74772b8f5de9c99eae87a not found: ID does not exist" Oct 29 23:04:59.078889 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:59.078855 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl"] Oct 29 23:04:59.091875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:59.091841 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d430df-6484d5895-n4zvl"] Oct 29 23:04:59.128332 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:59.128294 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx"] Oct 29 23:04:59.136913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:59.136878 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d430df-848f9fb454-vl9xx"] Oct 29 23:04:59.485152 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:59.485118 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e0999a-213d-4f93-bbd4-3c007ae1d7eb" path="/var/lib/kubelet/pods/16e0999a-213d-4f93-bbd4-3c007ae1d7eb/volumes" Oct 29 23:04:59.485648 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:04:59.485631 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d001db-e9ee-46aa-9475-5306f125d606" path="/var/lib/kubelet/pods/79d001db-e9ee-46aa-9475-5306f125d606/volumes" Oct 29 23:05:00.869064 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:00.845368 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" Oct 29 23:05:02.036130 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:02.036087 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652"] Oct 29 23:05:02.036594 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:02.036372 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" containerID="cri-o://2c6d619f9a8f0b35d19d25773e3f12ddd4a4fa2e80f12e49d749bf3ac4947af3" gracePeriod=30 Oct 29 23:05:02.985472 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:02.985434 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59"] Oct 29 23:05:02.986718 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:02.986684 5057 cpu_manager.go:414] "containerMap: removing container" podUID="16e0999a-213d-4f93-bbd4-3c007ae1d7eb" containerName="payload-pre-processing" Oct 29 23:05:02.986718 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:02.986710 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e0999a-213d-4f93-bbd4-3c007ae1d7eb" containerName="payload-pre-processing" Oct 29 23:05:02.986921 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:02.986735 5057 cpu_manager.go:414] "containerMap: removing container" podUID="79d001db-e9ee-46aa-9475-5306f125d606" containerName="payload-processing" Oct 29 23:05:02.986921 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:02.986744 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d001db-e9ee-46aa-9475-5306f125d606" containerName="payload-processing" Oct 29 23:05:02.987049 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:02.986933 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e0999a-213d-4f93-bbd4-3c007ae1d7eb" containerName="payload-pre-processing" Oct 29 23:05:02.987049 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:02.986950 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d001db-e9ee-46aa-9475-5306f125d606" containerName="payload-processing" Oct 29 23:05:02.992708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:02.992679 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:02.997698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:02.997671 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-envoy\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:02.997840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:02.997749 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdhg5\" (UniqueName: \"kubernetes.io/projected/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-kube-api-access-qdhg5\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:02.997840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:02.997822 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-workload-socket\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:02.998148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:02.997862 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-token\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:02.998148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:02.997916 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-podinfo\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:02.998148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:02.998063 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istiod-ca-cert\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:02.998148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:02.998122 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-workload-certs\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:02.998448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:02.998205 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-data\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:02.998448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:02.998238 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-credential-socket\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:03.036533 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.036490 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-57b28e-openshift-default-dockercfg-sjdh8\"" Oct 29 23:05:03.039294 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.039264 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59"] Oct 29 23:05:03.100339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.100283 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-data\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:03.100936 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.100349 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-credential-socket\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:03.100936 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.100424 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-envoy\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:03.100936 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.100526 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdhg5\" (UniqueName: \"kubernetes.io/projected/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-kube-api-access-qdhg5\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:03.100936 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.100641 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-workload-socket\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:03.100936 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.100698 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-token\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:03.100936 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.100760 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-podinfo\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:03.101966 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.101532 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istiod-ca-cert\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:03.101966 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.101631 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-workload-certs\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:03.102726 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.102255 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-workload-certs\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:03.102928 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.102904 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-data\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:03.103222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.103181 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-credential-socket\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:03.105323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.105073 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-workload-socket\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:03.106769 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.106731 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istiod-ca-cert\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:03.136297 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.136101 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-token\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:03.141774 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.141722 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdhg5\" (UniqueName: \"kubernetes.io/projected/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-kube-api-access-qdhg5\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:03.143818 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.143786 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-podinfo\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:03.144488 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.144463 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-envoy\") pod \"e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:03.385645 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:03.385556 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:03.937551 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:05:03.937484 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:05:04.090564 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.090525 5057 generic.go:356] "Generic (PLEG): container finished" podID="f525a668-448c-44ed-8fac-21db37da4c29" containerID="3a15c28171201f3e2e413b776c629c6caeccc5bd03f816270cf1b4fb9e0f80e0" exitCode=0 Oct 29 23:05:04.092170 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.090610 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" event={"ID":"f525a668-448c-44ed-8fac-21db37da4c29","Type":"ContainerDied","Data":"3a15c28171201f3e2e413b776c629c6caeccc5bd03f816270cf1b4fb9e0f80e0"} Oct 29 23:05:04.092170 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.091112 5057 scope.go:122] "RemoveContainer" containerID="3a15c28171201f3e2e413b776c629c6caeccc5bd03f816270cf1b4fb9e0f80e0" Oct 29 23:05:04.092720 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.092690 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:04.092782 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.092768 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:04.092840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.092827 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:04.165505 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.165419 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59"] Oct 29 23:05:04.174732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.174324 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:04.174732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.174402 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:04.174732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.174433 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:04.793562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.793532 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk"] Oct 29 23:05:04.797537 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.797516 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:04.842259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.842227 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk"] Oct 29 23:05:04.903004 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.902961 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-workload-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:04.903224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.903028 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istiod-ca-cert\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:04.903224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.903128 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-workload-certs\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:04.903224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.903160 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-credential-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:04.903399 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.903222 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-podinfo\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:04.903453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.903426 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcnx5\" (UniqueName: \"kubernetes.io/projected/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-kube-api-access-jcnx5\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:04.903530 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.903512 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-data\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:04.903598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.903580 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-envoy\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:04.903653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:04.903631 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-token\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:05.005292 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.005178 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-workload-certs\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:05.005486 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.005324 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-credential-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:05.005486 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.005359 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-podinfo\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:05.005622 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.005519 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcnx5\" (UniqueName: \"kubernetes.io/projected/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-kube-api-access-jcnx5\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:05.005622 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.005587 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-workload-certs\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:05.005746 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.005592 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-data\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:05.005746 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.005675 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-credential-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:05.005852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.005752 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-envoy\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:05.005852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.005806 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-token\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:05.005956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.005844 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-data\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:05.005998 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.005970 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-workload-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:05.006043 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.005998 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istiod-ca-cert\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:05.006307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.006288 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-workload-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:05.006613 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.006587 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istiod-ca-cert\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:05.007925 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.007908 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-envoy\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:05.008151 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.008133 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-podinfo\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:05.055046 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.054955 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-token\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:05.055674 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.055646 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcnx5\" (UniqueName: \"kubernetes.io/projected/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-kube-api-access-jcnx5\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:05.150225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.134548 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:05.282948 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.282905 5057 generic.go:356] "Generic (PLEG): container finished" podID="d728ed56-8f9a-4396-8386-7ce602570421" containerID="18425919557b62eb641feaa3a65c07a14d9d83af576c81ff516db8896d16151d" exitCode=0 Oct 29 23:05:05.283140 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.283114 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" event={"ID":"d728ed56-8f9a-4396-8386-7ce602570421","Type":"ContainerDied","Data":"18425919557b62eb641feaa3a65c07a14d9d83af576c81ff516db8896d16151d"} Oct 29 23:05:05.283795 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.283768 5057 scope.go:122] "RemoveContainer" containerID="18425919557b62eb641feaa3a65c07a14d9d83af576c81ff516db8896d16151d" Oct 29 23:05:05.287720 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.287693 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" event={"ID":"f525a668-448c-44ed-8fac-21db37da4c29","Type":"ContainerStarted","Data":"d9cb9b5f8f27dafff60b70ac0c64a0f3ff6446c7ba21c566e0ffe1d178fa4d18"} Oct 29 23:05:05.291171 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.291124 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:05.291319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.291298 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:05.291419 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.291397 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:05.297293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.297270 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" event={"ID":"fdd750a6-8487-4d03-9e52-4c6b6ef9133b","Type":"ContainerStarted","Data":"d88e4d7855571e464b282f6e02800d1afca39961e0b9350fcff4a656bc6a40c4"} Oct 29 23:05:05.298492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.298470 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" event={"ID":"fdd750a6-8487-4d03-9e52-4c6b6ef9133b","Type":"ContainerStarted","Data":"801ffe65ea06927ecaa90e25c84d7ad2533def64835c6797d51517c9ce4ea1f4"} Oct 29 23:05:05.387167 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.387096 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:05.396814 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.396722 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:05.472677 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.469246 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr"] Oct 29 23:05:05.472677 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.469571 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" podUID="e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" containerName="istio-proxy" containerID="cri-o://1dda5a8541a8a3d9c471f86ab24f64041d10795edfde8582cbb1943188ee4347" gracePeriod=30 Oct 29 23:05:05.488633 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.488572 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" podStartSLOduration=3.488552059 podStartE2EDuration="3.488552059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:05.465240508 +0000 UTC m=+1352.116739097" watchObservedRunningTime="2026-10-29 23:05:05.488552059 +0000 UTC m=+1352.140050590" Oct 29 23:05:05.507236 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.506180 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn"] Oct 29 23:05:05.507236 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.506266 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n"] Oct 29 23:05:05.507236 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.506302 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt"] Oct 29 23:05:05.507236 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.506490 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" podUID="dcda00c4-c4da-43c2-b232-3f7c877c3ef1" containerName="istio-proxy" containerID="cri-o://45375d4ca47662b5ce77543914f8d008765ac9af51ef60cf3b15195900806c0d" gracePeriod=30 Oct 29 23:05:05.507236 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.506746 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" podUID="93b619e2-1f43-4b7e-98e8-b83587057e45" containerName="istio-proxy" containerID="cri-o://7fb315758b2f371c0a427df37e500c0f438a1dca21eebf34d335c9f0ce7ea557" gracePeriod=30 Oct 29 23:05:05.507236 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.506923 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" podUID="10e45f5c-eae2-4876-ab74-f5949bea38e1" containerName="istio-proxy" containerID="cri-o://aed255caba5d36dca54e550ff1ef5a858606089be8558109a895a58ff3e290d9" gracePeriod=30 Oct 29 23:05:05.748794 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:05:05.742404 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod927fbba6_8dbc_4e71_81d9_de5b2859ba5d.slice/crio-7c4e2ac423f85eb57f15e89b541692b6c2d7cc3dfd30ef8df77049ebe2b380ab WatchSource:0}: Error finding container 7c4e2ac423f85eb57f15e89b541692b6c2d7cc3dfd30ef8df77049ebe2b380ab: Status 404 returned error can't find the container with id 7c4e2ac423f85eb57f15e89b541692b6c2d7cc3dfd30ef8df77049ebe2b380ab Oct 29 23:05:05.748794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.745064 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:05:05.748794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.746699 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:05.748794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.746798 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:05.748794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.746845 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:05.768833 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.750672 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 29 23:05:05.768833 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.750725 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" podUID="f525a668-448c-44ed-8fac-21db37da4c29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 29 23:05:05.792282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:05.781840 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk"] Oct 29 23:05:06.346210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:06.344684 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" event={"ID":"d728ed56-8f9a-4396-8386-7ce602570421","Type":"ContainerStarted","Data":"b8ace4e57b9d7bae23977abf99140415de27b909d27661a88eed3d25debf9dcc"} Oct 29 23:05:06.364231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:06.351354 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" event={"ID":"927fbba6-8dbc-4e71-81d9-de5b2859ba5d","Type":"ContainerStarted","Data":"b06c63c268e8707a35327eb1844c311f17243f2a54cb0457315c6bf72956b402"} Oct 29 23:05:06.364231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:06.351408 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" event={"ID":"927fbba6-8dbc-4e71-81d9-de5b2859ba5d","Type":"ContainerStarted","Data":"7c4e2ac423f85eb57f15e89b541692b6c2d7cc3dfd30ef8df77049ebe2b380ab"} Oct 29 23:05:06.364231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:06.353264 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:06.364231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:06.358405 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:05:06.433242 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:06.408302 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:05:06.595717 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:06.591736 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" podStartSLOduration=2.591713581 podStartE2EDuration="2.591713581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:06.499743228 +0000 UTC m=+1353.151241758" watchObservedRunningTime="2026-10-29 23:05:06.591713581 +0000 UTC m=+1353.243212112" Oct 29 23:05:06.682232 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:06.665054 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:05:06.723238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:06.708485 5057 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 29 23:05:06.804305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:06.804260 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" podUID="d728ed56-8f9a-4396-8386-7ce602570421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 29 23:05:06.804561 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:06.769525 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc" Oct 29 23:05:06.858266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:06.756226 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 29 23:05:06.931372 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:06.927032 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" podUID="f525a668-448c-44ed-8fac-21db37da4c29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 29 23:05:07.138281 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:07.135512 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:07.293916 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:07.293450 5057 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 23:05:07.293916 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:07.293514 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" podUID="927fbba6-8dbc-4e71-81d9-de5b2859ba5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 23:05:07.417761 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:07.416370 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:05:07.492218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:07.428243 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc"] Oct 29 23:05:07.554877 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:07.554848 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:07.675710 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:07.673519 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-ce5978-openshift-default-dockercfg-ndzrm\"" Oct 29 23:05:07.675710 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:07.673927 5057 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 29 23:05:07.675710 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:07.673972 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" podUID="d728ed56-8f9a-4396-8386-7ce602570421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 29 23:05:07.676005 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:07.675935 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 23:05:07.676005 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:07.675977 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 23:05:07.875561 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:07.875514 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-envoy\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.132668 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.132562 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-credential-socket\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.133034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.133004 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-data\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.133290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.133271 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7p85\" (UniqueName: \"kubernetes.io/projected/0cb218f5-5d38-4afb-9357-0b2db3da958e-kube-api-access-f7p85\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.133637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.133616 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-token\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.133813 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.133796 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cb218f5-5d38-4afb-9357-0b2db3da958e-istiod-ca-cert\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.134004 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.133987 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-workload-socket\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.134200 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.134168 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-workload-certs\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.134395 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.134376 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-podinfo\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.140209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:07.967156 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 29 23:05:08.140406 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.140376 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" podUID="f525a668-448c-44ed-8fac-21db37da4c29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 29 23:05:08.169115 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.166770 5057 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 23:05:08.169115 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.166841 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" podUID="927fbba6-8dbc-4e71-81d9-de5b2859ba5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 23:05:08.262154 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.261774 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-token\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.262154 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.261855 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cb218f5-5d38-4afb-9357-0b2db3da958e-istiod-ca-cert\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.262154 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.261911 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-workload-socket\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.262154 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.261977 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-workload-certs\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.262154 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.262039 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-podinfo\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.262605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.262174 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-envoy\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.262605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.262229 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-credential-socket\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.262605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.262338 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-data\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.262605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.262434 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7p85\" (UniqueName: \"kubernetes.io/projected/0cb218f5-5d38-4afb-9357-0b2db3da958e-kube-api-access-f7p85\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.281761 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.281726 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-workload-socket\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.310049 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.293173 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8" Oct 29 23:05:08.310049 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.293497 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc"] Oct 29 23:05:08.334225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.329787 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-workload-certs\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.335312 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.335277 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-credential-socket\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.335729 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.335701 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-podinfo\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.342213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.336003 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-data\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.367223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.354768 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7p85\" (UniqueName: \"kubernetes.io/projected/0cb218f5-5d38-4afb-9357-0b2db3da958e-kube-api-access-f7p85\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.384262 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.381693 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-envoy\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:08.398217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:08.386291 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cb218f5-5d38-4afb-9357-0b2db3da958e-istiod-ca-cert\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:09.116476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.091777 5057 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 29 23:05:09.217581 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.159575 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 23:05:09.217767 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.217521 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" podUID="d728ed56-8f9a-4396-8386-7ce602570421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 29 23:05:09.406616 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.406510 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:05:09.441214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.413602 5057 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 23:05:09.441214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.427251 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" podUID="927fbba6-8dbc-4e71-81d9-de5b2859ba5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 23:05:09.441214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.413644 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 29 23:05:09.441214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.427309 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" podUID="f525a668-448c-44ed-8fac-21db37da4c29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 29 23:05:09.467227 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.467179 5057 generic.go:356] "Generic (PLEG): container finished" podID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerID="2c6d619f9a8f0b35d19d25773e3f12ddd4a4fa2e80f12e49d749bf3ac4947af3" exitCode=0 Oct 29 23:05:09.496250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.467514 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" event={"ID":"a5c8812a-4547-46fc-ad21-ab74038a0fae","Type":"ContainerDied","Data":"2c6d619f9a8f0b35d19d25773e3f12ddd4a4fa2e80f12e49d749bf3ac4947af3"} Oct 29 23:05:09.517596 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.489177 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-token\") pod \"e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:09.517596 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.512943 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 23:05:09.674918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.659297 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-envoy\") pod \"a5c8812a-4547-46fc-ad21-ab74038a0fae\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " Oct 29 23:05:09.674918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.659473 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-token\" (UniqueName: \"kubernetes.io/projected/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-token\") pod \"a5c8812a-4547-46fc-ad21-ab74038a0fae\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " Oct 29 23:05:09.674918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.659529 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-podinfo\") pod \"a5c8812a-4547-46fc-ad21-ab74038a0fae\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " Oct 29 23:05:09.674918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.659601 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5c8812a-4547-46fc-ad21-ab74038a0fae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c8812a-4547-46fc-ad21-ab74038a0fae-istiod-ca-cert\") pod \"a5c8812a-4547-46fc-ad21-ab74038a0fae\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " Oct 29 23:05:09.674918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.660611 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-data\") pod \"a5c8812a-4547-46fc-ad21-ab74038a0fae\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " Oct 29 23:05:09.674918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.660673 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c8812a-4547-46fc-ad21-ab74038a0fae-kube-api-access-rvc5g\" (UniqueName: \"kubernetes.io/projected/a5c8812a-4547-46fc-ad21-ab74038a0fae-kube-api-access-rvc5g\") pod \"a5c8812a-4547-46fc-ad21-ab74038a0fae\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " Oct 29 23:05:09.674918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.660738 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-workload-socket\") pod \"a5c8812a-4547-46fc-ad21-ab74038a0fae\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " Oct 29 23:05:09.674918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.660813 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-workload-certs\") pod \"a5c8812a-4547-46fc-ad21-ab74038a0fae\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " Oct 29 23:05:09.674918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.660922 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-credential-socket\") pod \"a5c8812a-4547-46fc-ad21-ab74038a0fae\" (UID: \"a5c8812a-4547-46fc-ad21-ab74038a0fae\") " Oct 29 23:05:09.682466 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.682349 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:05:09.700250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.644167 5057 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 29 23:05:09.700581 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.700546 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" podUID="d728ed56-8f9a-4396-8386-7ce602570421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 29 23:05:09.701104 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.663579 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-credential-socket" pod "a5c8812a-4547-46fc-ad21-ab74038a0fae" (UID: "a5c8812a-4547-46fc-ad21-ab74038a0fae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:09.701104 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.684723 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-workload-socket" pod "a5c8812a-4547-46fc-ad21-ab74038a0fae" (UID: "a5c8812a-4547-46fc-ad21-ab74038a0fae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:09.701104 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.685056 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c8812a-4547-46fc-ad21-ab74038a0fae-istiod-ca-cert" pod "a5c8812a-4547-46fc-ad21-ab74038a0fae" (UID: "a5c8812a-4547-46fc-ad21-ab74038a0fae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:09.703591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.703561 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-data" pod "a5c8812a-4547-46fc-ad21-ab74038a0fae" (UID: "a5c8812a-4547-46fc-ad21-ab74038a0fae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:09.743615 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.741700 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 29 23:05:09.743615 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.741760 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" podUID="f525a668-448c-44ed-8fac-21db37da4c29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 29 23:05:09.781787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.781752 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-workload-certs" pod "a5c8812a-4547-46fc-ad21-ab74038a0fae" (UID: "a5c8812a-4547-46fc-ad21-ab74038a0fae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:09.789473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.789440 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:09.789473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.789475 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:09.789714 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.789490 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:09.789714 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.789506 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:09.789714 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.789521 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c8812a-4547-46fc-ad21-ab74038a0fae-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:09.793221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.789972 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:09.818964 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.818751 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-token" pod "a5c8812a-4547-46fc-ad21-ab74038a0fae" (UID: "a5c8812a-4547-46fc-ad21-ab74038a0fae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:09.990391 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.990354 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-envoy" pod "a5c8812a-4547-46fc-ad21-ab74038a0fae" (UID: "a5c8812a-4547-46fc-ad21-ab74038a0fae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:09.990799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.990755 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c8812a-4547-46fc-ad21-ab74038a0fae-kube-api-access-rvc5g" pod "a5c8812a-4547-46fc-ad21-ab74038a0fae" (UID: "a5c8812a-4547-46fc-ad21-ab74038a0fae"). InnerVolumeSpecName "kube-api-access-rvc5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:09.991013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:09.990994 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-podinfo" pod "a5c8812a-4547-46fc-ad21-ab74038a0fae" (UID: "a5c8812a-4547-46fc-ad21-ab74038a0fae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:05:10.190988 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:10.180875 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.190988 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:10.180914 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.190988 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:10.180927 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c8812a-4547-46fc-ad21-ab74038a0fae-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.190988 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:10.180940 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvc5g\" (UniqueName: \"kubernetes.io/projected/a5c8812a-4547-46fc-ad21-ab74038a0fae-kube-api-access-rvc5g\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.297312 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:10.296275 5057 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 23:05:10.297312 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:10.296349 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" podUID="927fbba6-8dbc-4e71-81d9-de5b2859ba5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 23:05:10.376244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:10.375326 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:05:10.406870 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:10.406824 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:05:10.417503 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:10.417459 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:05:10.432364 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:10.432292 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:05:10.435219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:10.432917 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:05:10.712404 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:10.708005 5057 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 29 23:05:10.712404 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:10.708058 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" podUID="d728ed56-8f9a-4396-8386-7ce602570421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 29 23:05:10.844016 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:10.843936 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 29 23:05:10.844260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:10.844232 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" podUID="f525a668-448c-44ed-8fac-21db37da4c29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 29 23:05:10.946816 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:10.946772 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" event={"ID":"a5c8812a-4547-46fc-ad21-ab74038a0fae","Type":"ContainerDied","Data":"d58aa59af5efe7aef3bd071476a1abdacb0cf20223ac6e0e704659aab7a07cf2"} Oct 29 23:05:10.947283 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:10.947252 5057 scope.go:122] "RemoveContainer" containerID="2c6d619f9a8f0b35d19d25773e3f12ddd4a4fa2e80f12e49d749bf3ac4947af3" Oct 29 23:05:10.955342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:10.955318 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652" Oct 29 23:05:11.168492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:11.168403 5057 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 23:05:11.196315 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:11.196267 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" podUID="927fbba6-8dbc-4e71-81d9-de5b2859ba5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 23:05:11.329448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:11.325479 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:05:11.405762 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:11.404630 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:05:11.445207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:11.440075 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 23:05:11.781272 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:11.679446 5057 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 29 23:05:11.781272 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:11.679518 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" podUID="d728ed56-8f9a-4396-8386-7ce602570421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 29 23:05:11.781272 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:11.764538 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 29 23:05:11.781272 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:11.764595 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" podUID="f525a668-448c-44ed-8fac-21db37da4c29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 29 23:05:11.997842 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:05:11.997807 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb218f5_5d38_4afb_9357_0b2db3da958e.slice/crio-b58f118d16b6f115e687405a00050c1413b0c4e699b8aeb0117f6065442d1e14 WatchSource:0}: Error finding container b58f118d16b6f115e687405a00050c1413b0c4e699b8aeb0117f6065442d1e14: Status 404 returned error can't find the container with id b58f118d16b6f115e687405a00050c1413b0c4e699b8aeb0117f6065442d1e14 Oct 29 23:05:12.061266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:12.033174 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:12.061971 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:12.061921 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:12.062220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:12.062172 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:12.159345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:12.159304 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:05:12.177672 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:12.177636 5057 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 23:05:12.177934 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:12.177907 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" podUID="927fbba6-8dbc-4e71-81d9-de5b2859ba5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 23:05:12.447741 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:12.425584 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:05:12.582261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:12.547026 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:05:12.683285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:12.653053 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:05:12.707441 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:12.703900 5057 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 29 23:05:12.707441 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:12.703959 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" podUID="d728ed56-8f9a-4396-8386-7ce602570421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 29 23:05:12.747392 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:12.747360 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 29 23:05:12.747660 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:12.747601 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" podUID="f525a668-448c-44ed-8fac-21db37da4c29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 29 23:05:12.831210 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:05:12.830541 5057 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.343s" Oct 29 23:05:12.831210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:12.830631 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:05:12.831210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:12.830696 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc"] Oct 29 23:05:12.841398 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:12.841309 5057 generic.go:356] "Generic (PLEG): container finished" podID="10e45f5c-eae2-4876-ab74-f5949bea38e1" containerID="aed255caba5d36dca54e550ff1ef5a858606089be8558109a895a58ff3e290d9" exitCode=0 Oct 29 23:05:12.841722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:12.841696 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" event={"ID":"10e45f5c-eae2-4876-ab74-f5949bea38e1","Type":"ContainerDied","Data":"aed255caba5d36dca54e550ff1ef5a858606089be8558109a895a58ff3e290d9"} Oct 29 23:05:12.841887 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:12.841871 5057 scope.go:122] "RemoveContainer" containerID="4f7b40747316cbceabd34ca9c5d18e21a65a2476fc2bac9f61cf0f662e15603f" Oct 29 23:05:13.032468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:13.028176 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652"] Oct 29 23:05:13.082368 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:13.082334 5057 generic.go:356] "Generic (PLEG): container finished" podID="93b619e2-1f43-4b7e-98e8-b83587057e45" containerID="7fb315758b2f371c0a427df37e500c0f438a1dca21eebf34d335c9f0ce7ea557" exitCode=0 Oct 29 23:05:13.082787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:13.082754 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" event={"ID":"93b619e2-1f43-4b7e-98e8-b83587057e45","Type":"ContainerDied","Data":"7fb315758b2f371c0a427df37e500c0f438a1dca21eebf34d335c9f0ce7ea557"} Oct 29 23:05:13.085229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:13.085205 5057 generic.go:356] "Generic (PLEG): container finished" podID="dcda00c4-c4da-43c2-b232-3f7c877c3ef1" containerID="45375d4ca47662b5ce77543914f8d008765ac9af51ef60cf3b15195900806c0d" exitCode=0 Oct 29 23:05:13.088214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:13.085408 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" event={"ID":"dcda00c4-c4da-43c2-b232-3f7c877c3ef1","Type":"ContainerDied","Data":"45375d4ca47662b5ce77543914f8d008765ac9af51ef60cf3b15195900806c0d"} Oct 29 23:05:13.094273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:13.094176 5057 generic.go:356] "Generic (PLEG): container finished" podID="e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" containerID="1dda5a8541a8a3d9c471f86ab24f64041d10795edfde8582cbb1943188ee4347" exitCode=0 Oct 29 23:05:13.094501 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:13.094479 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" event={"ID":"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b","Type":"ContainerDied","Data":"1dda5a8541a8a3d9c471f86ab24f64041d10795edfde8582cbb1943188ee4347"} Oct 29 23:05:13.211920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:13.207346 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gn652"] Oct 29 23:05:13.217912 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:13.217408 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:05:13.239244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:13.219901 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:05:13.292345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:13.283953 5057 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 23:05:13.292345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:13.284013 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" podUID="927fbba6-8dbc-4e71-81d9-de5b2859ba5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 23:05:13.433219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:13.378578 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:05:13.433219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:13.379778 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" event={"ID":"0cb218f5-5d38-4afb-9357-0b2db3da958e","Type":"ContainerStarted","Data":"b58f118d16b6f115e687405a00050c1413b0c4e699b8aeb0117f6065442d1e14"} Oct 29 23:05:13.615393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:13.611573 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 23:05:13.924509 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:13.795656 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" path="/var/lib/kubelet/pods/a5c8812a-4547-46fc-ad21-ab74038a0fae/volumes" Oct 29 23:05:13.924509 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:13.819912 5057 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 29 23:05:13.924509 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:13.819964 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" podUID="d728ed56-8f9a-4396-8386-7ce602570421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 29 23:05:14.201290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.175474 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:05:14.436295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.411421 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 29 23:05:14.436295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.411483 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" podUID="93b619e2-1f43-4b7e-98e8-b83587057e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 29 23:05:14.494065 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.494028 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts"] Oct 29 23:05:14.494626 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.494579 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" podUID="b39228ac-4129-460c-9a6c-42efc3a207bb" containerName="istio-proxy" containerID="cri-o://f87a21640e31ee1117dc1d85d1c04f7b10659a2fe5ea0d3975bbec5d19c8e402" gracePeriod=30 Oct 29 23:05:14.599668 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.598806 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:05:14.605823 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.605771 5057 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 23:05:14.606481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.606421 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" podUID="927fbba6-8dbc-4e71-81d9-de5b2859ba5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 23:05:14.607484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.607460 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" event={"ID":"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b","Type":"ContainerDied","Data":"dafe07869c8e2a215ff03e1b687471d60ecaa9824f881c2febb8411586d8df41"} Oct 29 23:05:14.607660 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.607643 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dafe07869c8e2a215ff03e1b687471d60ecaa9824f881c2febb8411586d8df41" Oct 29 23:05:14.622225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.615681 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:05:14.694252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.694157 5057 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 29 23:05:14.698339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.698300 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" podUID="d728ed56-8f9a-4396-8386-7ce602570421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 29 23:05:14.817445 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.817397 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:05:14.895239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.832545 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:05:14.990311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.973157 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:05:14.996348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.993246 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-kube-api-access-24nbs\" (UniqueName: \"kubernetes.io/projected/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-kube-api-access-24nbs\") pod \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " Oct 29 23:05:14.996574 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.996555 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-token\" (UniqueName: \"kubernetes.io/projected/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-token\") pod \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " Oct 29 23:05:14.996724 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.996706 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-envoy\") pod \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " Oct 29 23:05:14.996908 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.996892 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-workload-certs\") pod \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " Oct 29 23:05:14.997118 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.997101 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istiod-ca-cert\") pod \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " Oct 29 23:05:14.997278 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.997263 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-data\") pod \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " Oct 29 23:05:14.997401 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.997386 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-podinfo\") pod \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " Oct 29 23:05:14.997522 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.997509 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-workload-socket\") pod \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " Oct 29 23:05:14.997656 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.997642 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-credential-socket\") pod \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\" (UID: \"e15c4bc1-06b1-45e3-b6bf-5aaaf901594b\") " Oct 29 23:05:14.998503 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.998471 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istiod-ca-cert" pod "e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" (UID: "e15c4bc1-06b1-45e3-b6bf-5aaaf901594b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:14.998899 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.998854 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:14.999214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:14.999176 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-data" pod "e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" (UID: "e15c4bc1-06b1-45e3-b6bf-5aaaf901594b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:15.005042 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.004763 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-token" pod "e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" (UID: "e15c4bc1-06b1-45e3-b6bf-5aaaf901594b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:15.005164 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.005089 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:05:15.006249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.006223 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-workload-socket" pod "e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" (UID: "e15c4bc1-06b1-45e3-b6bf-5aaaf901594b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:15.006599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.006568 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-credential-socket" pod "e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" (UID: "e15c4bc1-06b1-45e3-b6bf-5aaaf901594b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:15.007117 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.007092 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-workload-certs" pod "e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" (UID: "e15c4bc1-06b1-45e3-b6bf-5aaaf901594b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:15.007438 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.007389 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-envoy" pod "e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" (UID: "e15c4bc1-06b1-45e3-b6bf-5aaaf901594b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:15.009549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.009446 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-kube-api-access-24nbs" pod "e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" (UID: "e15c4bc1-06b1-45e3-b6bf-5aaaf901594b"). InnerVolumeSpecName "kube-api-access-24nbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:15.009549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.009515 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-podinfo" pod "e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" (UID: "e15c4bc1-06b1-45e3-b6bf-5aaaf901594b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:05:15.099686 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.099654 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e45f5c-eae2-4876-ab74-f5949bea38e1-kube-api-access-5xb7n\" (UniqueName: \"kubernetes.io/projected/10e45f5c-eae2-4876-ab74-f5949bea38e1-kube-api-access-5xb7n\") pod \"10e45f5c-eae2-4876-ab74-f5949bea38e1\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " Oct 29 23:05:15.099938 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.099917 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-data\") pod \"10e45f5c-eae2-4876-ab74-f5949bea38e1\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " Oct 29 23:05:15.100149 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.100131 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-podinfo\") pod \"10e45f5c-eae2-4876-ab74-f5949bea38e1\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " Oct 29 23:05:15.100371 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.100353 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-workload-socket\") pod \"10e45f5c-eae2-4876-ab74-f5949bea38e1\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " Oct 29 23:05:15.100563 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.100549 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-envoy\") pod \"10e45f5c-eae2-4876-ab74-f5949bea38e1\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " Oct 29 23:05:15.100731 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.100717 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-workload-certs\") pod \"10e45f5c-eae2-4876-ab74-f5949bea38e1\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " Oct 29 23:05:15.100933 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.100921 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-token\" (UniqueName: \"kubernetes.io/projected/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-token\") pod \"10e45f5c-eae2-4876-ab74-f5949bea38e1\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " Oct 29 23:05:15.101086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.101073 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10e45f5c-eae2-4876-ab74-f5949bea38e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e45f5c-eae2-4876-ab74-f5949bea38e1-istiod-ca-cert\") pod \"10e45f5c-eae2-4876-ab74-f5949bea38e1\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " Oct 29 23:05:15.101234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.101220 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-credential-socket\") pod \"10e45f5c-eae2-4876-ab74-f5949bea38e1\" (UID: \"10e45f5c-eae2-4876-ab74-f5949bea38e1\") " Oct 29 23:05:15.102395 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.102377 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24nbs\" (UniqueName: \"kubernetes.io/projected/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-kube-api-access-24nbs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.102548 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.102536 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.102661 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.102650 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.103473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.103459 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.103688 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.103677 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.103791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.103781 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.104005 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.103993 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.104111 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.104097 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.104257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.103053 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-workload-certs" pod "10e45f5c-eae2-4876-ab74-f5949bea38e1" (UID: "10e45f5c-eae2-4876-ab74-f5949bea38e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:15.104611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.104591 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-workload-socket" pod "10e45f5c-eae2-4876-ab74-f5949bea38e1" (UID: "10e45f5c-eae2-4876-ab74-f5949bea38e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:15.107861 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.106756 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-envoy" pod "10e45f5c-eae2-4876-ab74-f5949bea38e1" (UID: "10e45f5c-eae2-4876-ab74-f5949bea38e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:15.107861 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.107091 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e45f5c-eae2-4876-ab74-f5949bea38e1-istiod-ca-cert" pod "10e45f5c-eae2-4876-ab74-f5949bea38e1" (UID: "10e45f5c-eae2-4876-ab74-f5949bea38e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:15.107861 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.107350 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-credential-socket" pod "10e45f5c-eae2-4876-ab74-f5949bea38e1" (UID: "10e45f5c-eae2-4876-ab74-f5949bea38e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:15.107861 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.107511 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-data" pod "10e45f5c-eae2-4876-ab74-f5949bea38e1" (UID: "10e45f5c-eae2-4876-ab74-f5949bea38e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:15.113573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.113463 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-podinfo" pod "10e45f5c-eae2-4876-ab74-f5949bea38e1" (UID: "10e45f5c-eae2-4876-ab74-f5949bea38e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:05:15.115351 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.115330 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e45f5c-eae2-4876-ab74-f5949bea38e1-kube-api-access-5xb7n" pod "10e45f5c-eae2-4876-ab74-f5949bea38e1" (UID: "10e45f5c-eae2-4876-ab74-f5949bea38e1"). InnerVolumeSpecName "kube-api-access-5xb7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:15.117051 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.117023 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-token" pod "10e45f5c-eae2-4876-ab74-f5949bea38e1" (UID: "10e45f5c-eae2-4876-ab74-f5949bea38e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:15.174759 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.158083 5057 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 23:05:15.175086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.175048 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" podUID="927fbba6-8dbc-4e71-81d9-de5b2859ba5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 23:05:15.194421 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.193756 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:05:15.208617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.207796 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-workload-socket\") pod \"93b619e2-1f43-4b7e-98e8-b83587057e45\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " Oct 29 23:05:15.208617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.207972 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-credential-socket\") pod \"93b619e2-1f43-4b7e-98e8-b83587057e45\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " Oct 29 23:05:15.208617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.208098 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-envoy\") pod \"93b619e2-1f43-4b7e-98e8-b83587057e45\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " Oct 29 23:05:15.208617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.208144 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93b619e2-1f43-4b7e-98e8-b83587057e45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b619e2-1f43-4b7e-98e8-b83587057e45-istiod-ca-cert\") pod \"93b619e2-1f43-4b7e-98e8-b83587057e45\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " Oct 29 23:05:15.209871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.209002 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-workload-certs\") pod \"93b619e2-1f43-4b7e-98e8-b83587057e45\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " Oct 29 23:05:15.209871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.209118 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b619e2-1f43-4b7e-98e8-b83587057e45-kube-api-access-c4gjf\" (UniqueName: \"kubernetes.io/projected/93b619e2-1f43-4b7e-98e8-b83587057e45-kube-api-access-c4gjf\") pod \"93b619e2-1f43-4b7e-98e8-b83587057e45\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.210204 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-token\" (UniqueName: \"kubernetes.io/projected/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-token\") pod \"93b619e2-1f43-4b7e-98e8-b83587057e45\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.210291 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-podinfo\") pod \"93b619e2-1f43-4b7e-98e8-b83587057e45\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.210470 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-data\") pod \"93b619e2-1f43-4b7e-98e8-b83587057e45\" (UID: \"93b619e2-1f43-4b7e-98e8-b83587057e45\") " Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.210865 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b619e2-1f43-4b7e-98e8-b83587057e45-istiod-ca-cert" pod "93b619e2-1f43-4b7e-98e8-b83587057e45" (UID: "93b619e2-1f43-4b7e-98e8-b83587057e45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.211381 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-credential-socket" pod "93b619e2-1f43-4b7e-98e8-b83587057e45" (UID: "93b619e2-1f43-4b7e-98e8-b83587057e45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.211847 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.213459 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-workload-certs" pod "93b619e2-1f43-4b7e-98e8-b83587057e45" (UID: "93b619e2-1f43-4b7e-98e8-b83587057e45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.214218 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-workload-socket" pod "93b619e2-1f43-4b7e-98e8-b83587057e45" (UID: "93b619e2-1f43-4b7e-98e8-b83587057e45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.214552 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-data" pod "93b619e2-1f43-4b7e-98e8-b83587057e45" (UID: "93b619e2-1f43-4b7e-98e8-b83587057e45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.216832 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.216864 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.216877 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.216889 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.216902 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.216914 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.216928 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.216940 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.216951 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b619e2-1f43-4b7e-98e8-b83587057e45-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.216963 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e45f5c-eae2-4876-ab74-f5949bea38e1-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.216975 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.216989 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.217005 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xb7n\" (UniqueName: \"kubernetes.io/projected/10e45f5c-eae2-4876-ab74-f5949bea38e1-kube-api-access-5xb7n\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.219975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.217017 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e45f5c-eae2-4876-ab74-f5949bea38e1-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.253913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.244775 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b619e2-1f43-4b7e-98e8-b83587057e45-kube-api-access-c4gjf" pod "93b619e2-1f43-4b7e-98e8-b83587057e45" (UID: "93b619e2-1f43-4b7e-98e8-b83587057e45"). InnerVolumeSpecName "kube-api-access-c4gjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:15.303597 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.303527 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-token" pod "93b619e2-1f43-4b7e-98e8-b83587057e45" (UID: "93b619e2-1f43-4b7e-98e8-b83587057e45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:15.307296 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.306054 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-envoy" pod "93b619e2-1f43-4b7e-98e8-b83587057e45" (UID: "93b619e2-1f43-4b7e-98e8-b83587057e45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:15.307296 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.306048 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-podinfo" pod "93b619e2-1f43-4b7e-98e8-b83587057e45" (UID: "93b619e2-1f43-4b7e-98e8-b83587057e45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:05:15.318075 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.318012 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-podinfo\") pod \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " Oct 29 23:05:15.318308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.318285 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-workload-socket\") pod \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " Oct 29 23:05:15.318409 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.318399 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-token\" (UniqueName: \"kubernetes.io/projected/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-token\") pod \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " Oct 29 23:05:15.318600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.318580 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-envoy\") pod \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " Oct 29 23:05:15.318888 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.318861 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istiod-ca-cert\") pod \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " Oct 29 23:05:15.319074 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.319052 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-workload-certs\") pod \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " Oct 29 23:05:15.319224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.319210 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-data\") pod \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " Oct 29 23:05:15.319390 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.319378 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-kube-api-access-r6g8q\" (UniqueName: \"kubernetes.io/projected/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-kube-api-access-r6g8q\") pod \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " Oct 29 23:05:15.319572 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.319558 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-credential-socket\") pod \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\" (UID: \"dcda00c4-c4da-43c2-b232-3f7c877c3ef1\") " Oct 29 23:05:15.320223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.320205 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.320350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.320334 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4gjf\" (UniqueName: \"kubernetes.io/projected/93b619e2-1f43-4b7e-98e8-b83587057e45-kube-api-access-c4gjf\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.320440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.320430 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.352376 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.320722 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-workload-socket" pod "dcda00c4-c4da-43c2-b232-3f7c877c3ef1" (UID: "dcda00c4-c4da-43c2-b232-3f7c877c3ef1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:15.370286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.320761 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-workload-certs" pod "dcda00c4-c4da-43c2-b232-3f7c877c3ef1" (UID: "dcda00c4-c4da-43c2-b232-3f7c877c3ef1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:15.370286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.320973 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istiod-ca-cert" pod "dcda00c4-c4da-43c2-b232-3f7c877c3ef1" (UID: "dcda00c4-c4da-43c2-b232-3f7c877c3ef1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:15.370286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.321144 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-data" pod "dcda00c4-c4da-43c2-b232-3f7c877c3ef1" (UID: "dcda00c4-c4da-43c2-b232-3f7c877c3ef1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:15.370286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.349146 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-credential-socket" pod "dcda00c4-c4da-43c2-b232-3f7c877c3ef1" (UID: "dcda00c4-c4da-43c2-b232-3f7c877c3ef1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:15.370286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.353573 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-kube-api-access-r6g8q" pod "dcda00c4-c4da-43c2-b232-3f7c877c3ef1" (UID: "dcda00c4-c4da-43c2-b232-3f7c877c3ef1"). InnerVolumeSpecName "kube-api-access-r6g8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:15.370286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.356833 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-podinfo" pod "dcda00c4-c4da-43c2-b232-3f7c877c3ef1" (UID: "dcda00c4-c4da-43c2-b232-3f7c877c3ef1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:05:15.370286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.352462 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b619e2-1f43-4b7e-98e8-b83587057e45-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.370286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.359202 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-envoy" pod "dcda00c4-c4da-43c2-b232-3f7c877c3ef1" (UID: "dcda00c4-c4da-43c2-b232-3f7c877c3ef1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:15.370940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.370419 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-token" pod "dcda00c4-c4da-43c2-b232-3f7c877c3ef1" (UID: "dcda00c4-c4da-43c2-b232-3f7c877c3ef1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:15.460768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.460726 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6g8q\" (UniqueName: \"kubernetes.io/projected/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-kube-api-access-r6g8q\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.460768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.460763 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.460768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.460773 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.460768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.460782 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.461153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.460790 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.461153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.460801 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.461153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.460810 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.461153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.460818 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.461153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.460826 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcda00c4-c4da-43c2-b232-3f7c877c3ef1-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:15.639648 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.639615 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:05:15.640884 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.640863 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:05:15.667747 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.667721 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" Oct 29 23:05:15.667925 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.667768 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn" event={"ID":"93b619e2-1f43-4b7e-98e8-b83587057e45","Type":"ContainerDied","Data":"26cde0e9e232a90a5a26ca9fec04e0bc5272cbcf70563010b18b6aa8de06b468"} Oct 29 23:05:15.667925 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.667845 5057 scope.go:122] "RemoveContainer" containerID="7fb315758b2f371c0a427df37e500c0f438a1dca21eebf34d335c9f0ce7ea557" Oct 29 23:05:15.670473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.669743 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" event={"ID":"dcda00c4-c4da-43c2-b232-3f7c877c3ef1","Type":"ContainerDied","Data":"50217c638ee4d4aa908489875d7e05c9c4f428071a0c33dfca4afd8e2af7ac8c"} Oct 29 23:05:15.670473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.669837 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt" Oct 29 23:05:15.674100 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.673734 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" event={"ID":"0cb218f5-5d38-4afb-9357-0b2db3da958e","Type":"ContainerStarted","Data":"73a3f54198ffc6a3e5f5d9850cb7f9890874877f74fee6af44d39047b93ae598"} Oct 29 23:05:15.675277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.675256 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" Oct 29 23:05:15.675388 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.675293 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n" event={"ID":"10e45f5c-eae2-4876-ab74-f5949bea38e1","Type":"ContainerDied","Data":"b5830b5cf2ddaf6a5a98ba380df36c772283b0e46833674813685f1ae32cc8c5"} Oct 29 23:05:15.675388 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.675265 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr" Oct 29 23:05:15.681570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.681553 5057 scope.go:122] "RemoveContainer" containerID="45375d4ca47662b5ce77543914f8d008765ac9af51ef60cf3b15195900806c0d" Oct 29 23:05:15.736601 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.736465 5057 scope.go:122] "RemoveContainer" containerID="aed255caba5d36dca54e550ff1ef5a858606089be8558109a895a58ff3e290d9" Oct 29 23:05:15.791976 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.791921 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" podStartSLOduration=8.791905263 podStartE2EDuration="8.791905263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:15.786798295 +0000 UTC m=+1362.438296827" watchObservedRunningTime="2026-10-29 23:05:15.791905263 +0000 UTC m=+1362.443403793" Oct 29 23:05:15.796110 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.796081 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:15.802118 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.802092 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:15.829526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.829496 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt"] Oct 29 23:05:15.835960 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.835930 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-jdflt"] Oct 29 23:05:15.874821 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.874789 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr"] Oct 29 23:05:15.879483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.879458 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8vknr"] Oct 29 23:05:15.915467 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.915425 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n"] Oct 29 23:05:15.920594 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.920563 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5kw5n"] Oct 29 23:05:15.959620 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.959581 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn"] Oct 29 23:05:15.963651 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:15.963620 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-78nkn"] Oct 29 23:05:16.008804 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.008493 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d"] Oct 29 23:05:16.009480 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.009459 5057 cpu_manager.go:414] "containerMap: removing container" podUID="93b619e2-1f43-4b7e-98e8-b83587057e45" containerName="istio-proxy" Oct 29 23:05:16.009480 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.009481 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b619e2-1f43-4b7e-98e8-b83587057e45" containerName="istio-proxy" Oct 29 23:05:16.009708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.009504 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" containerName="istio-proxy" Oct 29 23:05:16.009708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.009512 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" containerName="istio-proxy" Oct 29 23:05:16.009708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.009548 5057 cpu_manager.go:414] "containerMap: removing container" podUID="dcda00c4-c4da-43c2-b232-3f7c877c3ef1" containerName="istio-proxy" Oct 29 23:05:16.009708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.009553 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcda00c4-c4da-43c2-b232-3f7c877c3ef1" containerName="istio-proxy" Oct 29 23:05:16.009708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.009561 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" Oct 29 23:05:16.009708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.009570 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" Oct 29 23:05:16.009708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.009581 5057 cpu_manager.go:414] "containerMap: removing container" podUID="10e45f5c-eae2-4876-ab74-f5949bea38e1" containerName="istio-proxy" Oct 29 23:05:16.009708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.009588 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e45f5c-eae2-4876-ab74-f5949bea38e1" containerName="istio-proxy" Oct 29 23:05:16.009708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.009601 5057 cpu_manager.go:414] "containerMap: removing container" podUID="10e45f5c-eae2-4876-ab74-f5949bea38e1" containerName="istio-proxy" Oct 29 23:05:16.009708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.009606 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e45f5c-eae2-4876-ab74-f5949bea38e1" containerName="istio-proxy" Oct 29 23:05:16.010159 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.009728 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b619e2-1f43-4b7e-98e8-b83587057e45" containerName="istio-proxy" Oct 29 23:05:16.010159 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.009740 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" containerName="istio-proxy" Oct 29 23:05:16.010159 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.009751 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcda00c4-c4da-43c2-b232-3f7c877c3ef1" containerName="istio-proxy" Oct 29 23:05:16.010159 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.009763 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c8812a-4547-46fc-ad21-ab74038a0fae" containerName="istio-proxy" Oct 29 23:05:16.010159 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.009769 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e45f5c-eae2-4876-ab74-f5949bea38e1" containerName="istio-proxy" Oct 29 23:05:16.010159 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.009775 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e45f5c-eae2-4876-ab74-f5949bea38e1" containerName="istio-proxy" Oct 29 23:05:16.014969 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.014882 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.037549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.033477 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-dfedfe-openshift-default-dockercfg-6kkpd\"" Oct 29 23:05:16.045508 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.044764 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d"] Oct 29 23:05:16.177363 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.173720 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-workload-socket\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.181402 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.181365 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-token\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.181755 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.181731 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbpqn\" (UniqueName: \"kubernetes.io/projected/bec61ff1-da16-48fd-97cc-7a6cc85213e3-kube-api-access-vbpqn\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.204353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.204313 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-workload-certs\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.204663 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.204641 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-podinfo\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.204779 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.204765 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istiod-ca-cert\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.204937 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.204923 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-credential-socket\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.205063 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.205048 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-envoy\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.205181 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.205168 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-data\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.311378 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.311283 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-workload-socket\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.312588 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.312565 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-token\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.313210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.313169 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbpqn\" (UniqueName: \"kubernetes.io/projected/bec61ff1-da16-48fd-97cc-7a6cc85213e3-kube-api-access-vbpqn\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.313658 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.313638 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-workload-certs\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.314123 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.314105 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-podinfo\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.323625 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.323599 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istiod-ca-cert\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.344746 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.344714 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-credential-socket\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.345039 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.345013 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-envoy\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.345227 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.345209 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-data\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.345525 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.314008 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-workload-certs\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.345620 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.312411 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-workload-socket\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.345687 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.345536 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-credential-socket\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.346034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.324450 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istiod-ca-cert\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.346137 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.345987 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-data\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.397000 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.396725 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-token\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.444301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.407899 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbpqn\" (UniqueName: \"kubernetes.io/projected/bec61ff1-da16-48fd-97cc-7a6cc85213e3-kube-api-access-vbpqn\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.444301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.408452 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-podinfo\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.444301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.421010 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:16.444301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.425241 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:16.465207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.454586 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-envoy\") pod \"e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.465903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.465879 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:05:16.684714 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.684681 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:16.831241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.829633 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:16.831241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:16.830827 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:05:17.213077 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:17.213047 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d"] Oct 29 23:05:17.221703 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:17.221658 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:17.221823 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:17.221750 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:17.221823 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:17.221793 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:17.484142 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:17.484054 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10e45f5c-eae2-4876-ab74-f5949bea38e1" path="/var/lib/kubelet/pods/10e45f5c-eae2-4876-ab74-f5949bea38e1/volumes" Oct 29 23:05:17.484937 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:17.484908 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93b619e2-1f43-4b7e-98e8-b83587057e45" path="/var/lib/kubelet/pods/93b619e2-1f43-4b7e-98e8-b83587057e45/volumes" Oct 29 23:05:17.485668 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:17.485643 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcda00c4-c4da-43c2-b232-3f7c877c3ef1" path="/var/lib/kubelet/pods/dcda00c4-c4da-43c2-b232-3f7c877c3ef1/volumes" Oct 29 23:05:17.486769 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:17.486744 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e15c4bc1-06b1-45e3-b6bf-5aaaf901594b" path="/var/lib/kubelet/pods/e15c4bc1-06b1-45e3-b6bf-5aaaf901594b/volumes" Oct 29 23:05:17.840172 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:17.840068 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" event={"ID":"bec61ff1-da16-48fd-97cc-7a6cc85213e3","Type":"ContainerStarted","Data":"d97a49bf83f1840cf074f16f1e08c6855c6c11f810cc19211ea2be9eb73e3334"} Oct 29 23:05:17.840172 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:17.840129 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" event={"ID":"bec61ff1-da16-48fd-97cc-7a6cc85213e3","Type":"ContainerStarted","Data":"de5994c7299bc7382d29d08c681ec982a5d113ba6b7df72fb17bdf80c86eaf18"} Oct 29 23:05:17.904305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:17.904244 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" podStartSLOduration=2.904229182 podStartE2EDuration="2.904229182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:17.902138386 +0000 UTC m=+1364.553636917" watchObservedRunningTime="2026-10-29 23:05:17.904229182 +0000 UTC m=+1364.555727711" Oct 29 23:05:18.686949 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:18.686877 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:18.695374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:18.695320 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:18.848822 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:18.848789 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:18.849473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:18.849452 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:05:20.245684 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.245656 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:05:20.295226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.295180 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39228ac-4129-460c-9a6c-42efc3a207bb-kube-api-access-qxr4f\" (UniqueName: \"kubernetes.io/projected/b39228ac-4129-460c-9a6c-42efc3a207bb-kube-api-access-qxr4f\") pod \"b39228ac-4129-460c-9a6c-42efc3a207bb\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " Oct 29 23:05:20.295416 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.295299 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-envoy\") pod \"b39228ac-4129-460c-9a6c-42efc3a207bb\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " Oct 29 23:05:20.295416 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.295355 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-podinfo\") pod \"b39228ac-4129-460c-9a6c-42efc3a207bb\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " Oct 29 23:05:20.295551 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.295456 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-credential-socket\") pod \"b39228ac-4129-460c-9a6c-42efc3a207bb\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " Oct 29 23:05:20.295551 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.295495 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39228ac-4129-460c-9a6c-42efc3a207bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39228ac-4129-460c-9a6c-42efc3a207bb-istiod-ca-cert\") pod \"b39228ac-4129-460c-9a6c-42efc3a207bb\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " Oct 29 23:05:20.295665 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.295609 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-token\" (UniqueName: \"kubernetes.io/projected/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-token\") pod \"b39228ac-4129-460c-9a6c-42efc3a207bb\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " Oct 29 23:05:20.295719 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.295685 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-workload-certs\") pod \"b39228ac-4129-460c-9a6c-42efc3a207bb\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " Oct 29 23:05:20.295787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.295770 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-workload-socket\") pod \"b39228ac-4129-460c-9a6c-42efc3a207bb\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " Oct 29 23:05:20.295842 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.295778 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-credential-socket" pod "b39228ac-4129-460c-9a6c-42efc3a207bb" (UID: "b39228ac-4129-460c-9a6c-42efc3a207bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:20.295842 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.295834 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-data\") pod \"b39228ac-4129-460c-9a6c-42efc3a207bb\" (UID: \"b39228ac-4129-460c-9a6c-42efc3a207bb\") " Oct 29 23:05:20.296015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.295991 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39228ac-4129-460c-9a6c-42efc3a207bb-istiod-ca-cert" pod "b39228ac-4129-460c-9a6c-42efc3a207bb" (UID: "b39228ac-4129-460c-9a6c-42efc3a207bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:20.296296 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.296271 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-workload-certs" pod "b39228ac-4129-460c-9a6c-42efc3a207bb" (UID: "b39228ac-4129-460c-9a6c-42efc3a207bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:20.296650 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.296626 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-workload-socket" pod "b39228ac-4129-460c-9a6c-42efc3a207bb" (UID: "b39228ac-4129-460c-9a6c-42efc3a207bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:20.296799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.296682 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-data" pod "b39228ac-4129-460c-9a6c-42efc3a207bb" (UID: "b39228ac-4129-460c-9a6c-42efc3a207bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:20.296799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.296749 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:20.296799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.296768 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39228ac-4129-460c-9a6c-42efc3a207bb-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:20.296799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.296781 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:20.298294 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.298269 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-envoy" pod "b39228ac-4129-460c-9a6c-42efc3a207bb" (UID: "b39228ac-4129-460c-9a6c-42efc3a207bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:20.298549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.298519 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-token" pod "b39228ac-4129-460c-9a6c-42efc3a207bb" (UID: "b39228ac-4129-460c-9a6c-42efc3a207bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:20.298646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.298576 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39228ac-4129-460c-9a6c-42efc3a207bb-kube-api-access-qxr4f" pod "b39228ac-4129-460c-9a6c-42efc3a207bb" (UID: "b39228ac-4129-460c-9a6c-42efc3a207bb"). InnerVolumeSpecName "kube-api-access-qxr4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:20.298646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.298614 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-podinfo" pod "b39228ac-4129-460c-9a6c-42efc3a207bb" (UID: "b39228ac-4129-460c-9a6c-42efc3a207bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:05:20.336882 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.336842 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4"] Oct 29 23:05:20.338098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.338070 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b39228ac-4129-460c-9a6c-42efc3a207bb" containerName="istio-proxy" Oct 29 23:05:20.338098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.338098 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39228ac-4129-460c-9a6c-42efc3a207bb" containerName="istio-proxy" Oct 29 23:05:20.338306 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.338146 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b39228ac-4129-460c-9a6c-42efc3a207bb" containerName="istio-proxy" Oct 29 23:05:20.338306 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.338153 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39228ac-4129-460c-9a6c-42efc3a207bb" containerName="istio-proxy" Oct 29 23:05:20.338413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.338388 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39228ac-4129-460c-9a6c-42efc3a207bb" containerName="istio-proxy" Oct 29 23:05:20.338413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.338410 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39228ac-4129-460c-9a6c-42efc3a207bb" containerName="istio-proxy" Oct 29 23:05:20.342692 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.342668 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4" Oct 29 23:05:20.361882 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.361824 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4"] Oct 29 23:05:20.362772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.362747 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-ce5978-dockercfg-sl6j9\"" Oct 29 23:05:20.450582 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.410461 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-ce5978\"" Oct 29 23:05:20.450582 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.412000 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:20.450582 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.412027 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:20.450582 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.412041 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxr4f\" (UniqueName: \"kubernetes.io/projected/b39228ac-4129-460c-9a6c-42efc3a207bb-kube-api-access-qxr4f\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:20.450582 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.412057 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:20.450582 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.412071 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:20.450582 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.412082 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39228ac-4129-460c-9a6c-42efc3a207bb-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:05:20.577212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.547228 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3473cd-60a2-4f67-9da9-6ed27a733f44-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4\" (UID: \"dd3473cd-60a2-4f67-9da9-6ed27a733f44\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4" Oct 29 23:05:20.692214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.666344 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnlk9\" (UniqueName: \"kubernetes.io/projected/dd3473cd-60a2-4f67-9da9-6ed27a733f44-kube-api-access-jnlk9\") pod \"payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4\" (UID: \"dd3473cd-60a2-4f67-9da9-6ed27a733f44\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4" Oct 29 23:05:20.775259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.773304 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3473cd-60a2-4f67-9da9-6ed27a733f44-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4\" (UID: \"dd3473cd-60a2-4f67-9da9-6ed27a733f44\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4" Oct 29 23:05:20.775259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.773389 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnlk9\" (UniqueName: \"kubernetes.io/projected/dd3473cd-60a2-4f67-9da9-6ed27a733f44-kube-api-access-jnlk9\") pod \"payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4\" (UID: \"dd3473cd-60a2-4f67-9da9-6ed27a733f44\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4" Oct 29 23:05:20.775259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.775043 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3473cd-60a2-4f67-9da9-6ed27a733f44-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4\" (UID: \"dd3473cd-60a2-4f67-9da9-6ed27a733f44\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4" Oct 29 23:05:20.844208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.837368 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnlk9\" (UniqueName: \"kubernetes.io/projected/dd3473cd-60a2-4f67-9da9-6ed27a733f44-kube-api-access-jnlk9\") pod \"payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4\" (UID: \"dd3473cd-60a2-4f67-9da9-6ed27a733f44\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4" Oct 29 23:05:20.873953 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.873919 5057 generic.go:356] "Generic (PLEG): container finished" podID="b39228ac-4129-460c-9a6c-42efc3a207bb" containerID="f87a21640e31ee1117dc1d85d1c04f7b10659a2fe5ea0d3975bbec5d19c8e402" exitCode=0 Oct 29 23:05:20.874891 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.874868 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" Oct 29 23:05:20.876453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.876425 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" event={"ID":"b39228ac-4129-460c-9a6c-42efc3a207bb","Type":"ContainerDied","Data":"f87a21640e31ee1117dc1d85d1c04f7b10659a2fe5ea0d3975bbec5d19c8e402"} Oct 29 23:05:20.876553 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.876485 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts" event={"ID":"b39228ac-4129-460c-9a6c-42efc3a207bb","Type":"ContainerDied","Data":"654a3a762912fab726c2c8c230dafc892ce74c506e3e3ac20cf180d13d186d27"} Oct 29 23:05:20.876553 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.876515 5057 scope.go:122] "RemoveContainer" containerID="f87a21640e31ee1117dc1d85d1c04f7b10659a2fe5ea0d3975bbec5d19c8e402" Oct 29 23:05:20.891125 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.891096 5057 scope.go:122] "RemoveContainer" containerID="b76a1dead114db83f75eb9643df5f0ed2b3bcf297a1746b64661dd1ebfe04460" Oct 29 23:05:20.905750 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.905726 5057 scope.go:122] "RemoveContainer" containerID="f87a21640e31ee1117dc1d85d1c04f7b10659a2fe5ea0d3975bbec5d19c8e402" Oct 29 23:05:20.906071 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:05:20.906053 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f87a21640e31ee1117dc1d85d1c04f7b10659a2fe5ea0d3975bbec5d19c8e402\": container with ID starting with f87a21640e31ee1117dc1d85d1c04f7b10659a2fe5ea0d3975bbec5d19c8e402 not found: ID does not exist" containerID="f87a21640e31ee1117dc1d85d1c04f7b10659a2fe5ea0d3975bbec5d19c8e402" Oct 29 23:05:20.906111 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.906078 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f87a21640e31ee1117dc1d85d1c04f7b10659a2fe5ea0d3975bbec5d19c8e402"} err="failed to get container status \"f87a21640e31ee1117dc1d85d1c04f7b10659a2fe5ea0d3975bbec5d19c8e402\": rpc error: code = NotFound desc = could not find container \"f87a21640e31ee1117dc1d85d1c04f7b10659a2fe5ea0d3975bbec5d19c8e402\": container with ID starting with f87a21640e31ee1117dc1d85d1c04f7b10659a2fe5ea0d3975bbec5d19c8e402 not found: ID does not exist" Oct 29 23:05:20.906111 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.906096 5057 scope.go:122] "RemoveContainer" containerID="b76a1dead114db83f75eb9643df5f0ed2b3bcf297a1746b64661dd1ebfe04460" Oct 29 23:05:20.906475 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:05:20.906452 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b76a1dead114db83f75eb9643df5f0ed2b3bcf297a1746b64661dd1ebfe04460\": container with ID starting with b76a1dead114db83f75eb9643df5f0ed2b3bcf297a1746b64661dd1ebfe04460 not found: ID does not exist" containerID="b76a1dead114db83f75eb9643df5f0ed2b3bcf297a1746b64661dd1ebfe04460" Oct 29 23:05:20.906563 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.906480 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b76a1dead114db83f75eb9643df5f0ed2b3bcf297a1746b64661dd1ebfe04460"} err="failed to get container status \"b76a1dead114db83f75eb9643df5f0ed2b3bcf297a1746b64661dd1ebfe04460\": rpc error: code = NotFound desc = could not find container \"b76a1dead114db83f75eb9643df5f0ed2b3bcf297a1746b64661dd1ebfe04460\": container with ID starting with b76a1dead114db83f75eb9643df5f0ed2b3bcf297a1746b64661dd1ebfe04460 not found: ID does not exist" Oct 29 23:05:20.941918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.941881 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts"] Oct 29 23:05:20.958147 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.958114 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7c486c-openshift-default-944cf4797-wc2ts"] Oct 29 23:05:20.979434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:20.979408 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4" Oct 29 23:05:21.289591 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:05:21.289526 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3473cd_60a2_4f67_9da9_6ed27a733f44.slice/crio-72f44c60f8dc4b065643c17ad1f6f50053deba8f638fd7592d9ceeeda194b3d5 WatchSource:0}: Error finding container 72f44c60f8dc4b065643c17ad1f6f50053deba8f638fd7592d9ceeeda194b3d5: Status 404 returned error can't find the container with id 72f44c60f8dc4b065643c17ad1f6f50053deba8f638fd7592d9ceeeda194b3d5 Oct 29 23:05:21.289996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:21.289577 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4"] Oct 29 23:05:21.485625 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:21.485592 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39228ac-4129-460c-9a6c-42efc3a207bb" path="/var/lib/kubelet/pods/b39228ac-4129-460c-9a6c-42efc3a207bb/volumes" Oct 29 23:05:21.884544 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:21.884511 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4" event={"ID":"dd3473cd-60a2-4f67-9da9-6ed27a733f44","Type":"ContainerStarted","Data":"985ffd8ded60a0345bafce53d688f14b83f38d9e5fda113021c4a19f7bf4d7c9"} Oct 29 23:05:21.884733 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:21.884561 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4" event={"ID":"dd3473cd-60a2-4f67-9da9-6ed27a733f44","Type":"ContainerStarted","Data":"72f44c60f8dc4b065643c17ad1f6f50053deba8f638fd7592d9ceeeda194b3d5"} Oct 29 23:05:21.884807 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:21.884728 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4" Oct 29 23:05:21.932525 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:21.932456 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4" podStartSLOduration=1.9324380300000001 podStartE2EDuration="1.93243803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:21.92927395 +0000 UTC m=+1368.580772483" watchObservedRunningTime="2026-10-29 23:05:21.93243803 +0000 UTC m=+1368.583936560" Oct 29 23:05:22.761302 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:22.761266 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5"] Oct 29 23:05:22.765469 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:22.765450 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5" Oct 29 23:05:22.786076 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:22.786050 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-dfedfe\"" Oct 29 23:05:22.787940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:22.787902 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5"] Oct 29 23:05:22.796492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:22.796460 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-dfedfe-dockercfg-tphbm\"" Oct 29 23:05:22.866209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:22.842694 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx7wg\" (UniqueName: \"kubernetes.io/projected/f486535b-541f-4e2a-a5ba-5dfbb9314059-kube-api-access-qx7wg\") pod \"payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5\" (UID: \"f486535b-541f-4e2a-a5ba-5dfbb9314059\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5" Oct 29 23:05:22.866209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:22.842810 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f486535b-541f-4e2a-a5ba-5dfbb9314059-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5\" (UID: \"f486535b-541f-4e2a-a5ba-5dfbb9314059\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5" Oct 29 23:05:23.159353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:23.127477 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f486535b-541f-4e2a-a5ba-5dfbb9314059-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5\" (UID: \"f486535b-541f-4e2a-a5ba-5dfbb9314059\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5" Oct 29 23:05:23.159353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:23.128102 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx7wg\" (UniqueName: \"kubernetes.io/projected/f486535b-541f-4e2a-a5ba-5dfbb9314059-kube-api-access-qx7wg\") pod \"payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5\" (UID: \"f486535b-541f-4e2a-a5ba-5dfbb9314059\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5" Oct 29 23:05:23.159353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:23.149907 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f486535b-541f-4e2a-a5ba-5dfbb9314059-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5\" (UID: \"f486535b-541f-4e2a-a5ba-5dfbb9314059\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5" Oct 29 23:05:23.373256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:23.371445 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx7wg\" (UniqueName: \"kubernetes.io/projected/f486535b-541f-4e2a-a5ba-5dfbb9314059-kube-api-access-qx7wg\") pod \"payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5\" (UID: \"f486535b-541f-4e2a-a5ba-5dfbb9314059\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5" Oct 29 23:05:23.490153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:23.490102 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5" Oct 29 23:05:24.104215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.104119 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv"] Oct 29 23:05:24.392654 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.392577 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv"] Oct 29 23:05:24.398379 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.393321 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v"] Oct 29 23:05:24.398379 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.394041 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv" Oct 29 23:05:24.401853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.401817 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v"] Oct 29 23:05:24.402024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.402009 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v" Oct 29 23:05:24.428890 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.428862 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7af1a5-dockercfg-f2dbn\"" Oct 29 23:05:24.429034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.428988 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7af1a5\"" Oct 29 23:05:24.486984 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.486204 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sthjc\" (UniqueName: \"kubernetes.io/projected/e7897e3b-8c0e-45be-a3b1-ad5ccf299283-kube-api-access-sthjc\") pod \"payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv\" (UID: \"e7897e3b-8c0e-45be-a3b1-ad5ccf299283\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv" Oct 29 23:05:24.486984 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.486278 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng5q9\" (UniqueName: \"kubernetes.io/projected/cdf4f520-c1fb-4178-9685-2ddf12262613-kube-api-access-ng5q9\") pod \"payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v\" (UID: \"cdf4f520-c1fb-4178-9685-2ddf12262613\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v" Oct 29 23:05:24.486984 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.486429 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf4f520-c1fb-4178-9685-2ddf12262613-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v\" (UID: \"cdf4f520-c1fb-4178-9685-2ddf12262613\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v" Oct 29 23:05:24.486984 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.486460 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7897e3b-8c0e-45be-a3b1-ad5ccf299283-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv\" (UID: \"e7897e3b-8c0e-45be-a3b1-ad5ccf299283\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv" Oct 29 23:05:24.604440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.604324 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf4f520-c1fb-4178-9685-2ddf12262613-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v\" (UID: \"cdf4f520-c1fb-4178-9685-2ddf12262613\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v" Oct 29 23:05:24.604440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.604393 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7897e3b-8c0e-45be-a3b1-ad5ccf299283-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv\" (UID: \"e7897e3b-8c0e-45be-a3b1-ad5ccf299283\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv" Oct 29 23:05:24.605120 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.605075 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sthjc\" (UniqueName: \"kubernetes.io/projected/e7897e3b-8c0e-45be-a3b1-ad5ccf299283-kube-api-access-sthjc\") pod \"payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv\" (UID: \"e7897e3b-8c0e-45be-a3b1-ad5ccf299283\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv" Oct 29 23:05:24.605514 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.605401 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng5q9\" (UniqueName: \"kubernetes.io/projected/cdf4f520-c1fb-4178-9685-2ddf12262613-kube-api-access-ng5q9\") pod \"payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v\" (UID: \"cdf4f520-c1fb-4178-9685-2ddf12262613\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v" Oct 29 23:05:24.607489 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.607469 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf4f520-c1fb-4178-9685-2ddf12262613-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v\" (UID: \"cdf4f520-c1fb-4178-9685-2ddf12262613\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v" Oct 29 23:05:24.639223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.637030 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7897e3b-8c0e-45be-a3b1-ad5ccf299283-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv\" (UID: \"e7897e3b-8c0e-45be-a3b1-ad5ccf299283\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv" Oct 29 23:05:24.661271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.660777 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng5q9\" (UniqueName: \"kubernetes.io/projected/cdf4f520-c1fb-4178-9685-2ddf12262613-kube-api-access-ng5q9\") pod \"payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v\" (UID: \"cdf4f520-c1fb-4178-9685-2ddf12262613\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v" Oct 29 23:05:24.667218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.664897 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sthjc\" (UniqueName: \"kubernetes.io/projected/e7897e3b-8c0e-45be-a3b1-ad5ccf299283-kube-api-access-sthjc\") pod \"payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv\" (UID: \"e7897e3b-8c0e-45be-a3b1-ad5ccf299283\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv" Oct 29 23:05:24.698163 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:05:24.698131 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf486535b_541f_4e2a_a5ba_5dfbb9314059.slice/crio-c1501007ea0292ce68245c6fde798ba111c74588e60a78331ffdafc1ce9422a7 WatchSource:0}: Error finding container c1501007ea0292ce68245c6fde798ba111c74588e60a78331ffdafc1ce9422a7: Status 404 returned error can't find the container with id c1501007ea0292ce68245c6fde798ba111c74588e60a78331ffdafc1ce9422a7 Oct 29 23:05:24.699382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.699354 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5"] Oct 29 23:05:24.720311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.720287 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv" Oct 29 23:05:24.741054 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:24.741021 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v" Oct 29 23:05:25.041229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:25.041167 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv"] Oct 29 23:05:25.042148 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:05:25.042118 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7897e3b_8c0e_45be_a3b1_ad5ccf299283.slice/crio-4b03feb3eea734abdaf607db81871e75d4537649dbafce70d728f9cf7965b694 WatchSource:0}: Error finding container 4b03feb3eea734abdaf607db81871e75d4537649dbafce70d728f9cf7965b694: Status 404 returned error can't find the container with id 4b03feb3eea734abdaf607db81871e75d4537649dbafce70d728f9cf7965b694 Oct 29 23:05:25.082075 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:25.082045 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v"] Oct 29 23:05:25.084673 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:05:25.084647 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf4f520_c1fb_4178_9685_2ddf12262613.slice/crio-6e6194d2a2cef3315fccb004c55269e1b02f121d31172278f7a34365e11f931e WatchSource:0}: Error finding container 6e6194d2a2cef3315fccb004c55269e1b02f121d31172278f7a34365e11f931e: Status 404 returned error can't find the container with id 6e6194d2a2cef3315fccb004c55269e1b02f121d31172278f7a34365e11f931e Oct 29 23:05:25.414945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:25.414903 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5" event={"ID":"f486535b-541f-4e2a-a5ba-5dfbb9314059","Type":"ContainerStarted","Data":"d0be8ab50080a354810fc94c7f01325f7a476a93405e4b068fff44f074e479d8"} Oct 29 23:05:25.422685 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:25.422642 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5" event={"ID":"f486535b-541f-4e2a-a5ba-5dfbb9314059","Type":"ContainerStarted","Data":"c1501007ea0292ce68245c6fde798ba111c74588e60a78331ffdafc1ce9422a7"} Oct 29 23:05:25.422829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:25.422700 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv" event={"ID":"e7897e3b-8c0e-45be-a3b1-ad5ccf299283","Type":"ContainerStarted","Data":"2ce97c606bddb4fd86fd6f99b161c90cbc5bdd8c34e6293e4c2530bbc2a1c8c0"} Oct 29 23:05:25.422829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:25.422729 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv" event={"ID":"e7897e3b-8c0e-45be-a3b1-ad5ccf299283","Type":"ContainerStarted","Data":"4b03feb3eea734abdaf607db81871e75d4537649dbafce70d728f9cf7965b694"} Oct 29 23:05:25.422829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:25.422765 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5" Oct 29 23:05:25.422829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:25.422795 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv" Oct 29 23:05:25.428746 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:25.428711 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v" event={"ID":"cdf4f520-c1fb-4178-9685-2ddf12262613","Type":"ContainerStarted","Data":"ad6254ddc3c11b7ed79a940383ddcca5fc3594f5b44a9e79bd2695f8cbffa7d5"} Oct 29 23:05:25.431396 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:25.428770 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v" event={"ID":"cdf4f520-c1fb-4178-9685-2ddf12262613","Type":"ContainerStarted","Data":"6e6194d2a2cef3315fccb004c55269e1b02f121d31172278f7a34365e11f931e"} Oct 29 23:05:25.431396 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:25.429587 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v" Oct 29 23:05:25.469089 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:25.468599 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5" podStartSLOduration=3.468580265 podStartE2EDuration="3.468580265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:25.463046899 +0000 UTC m=+1372.114545430" watchObservedRunningTime="2026-10-29 23:05:25.468580265 +0000 UTC m=+1372.120078800" Oct 29 23:05:25.512039 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:25.511985 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v" podStartSLOduration=1.511970572 podStartE2EDuration="1.511970572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:25.507708809 +0000 UTC m=+1372.159207342" watchObservedRunningTime="2026-10-29 23:05:25.511970572 +0000 UTC m=+1372.163469103" Oct 29 23:05:25.555449 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:25.555390 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv" podStartSLOduration=2.555372243 podStartE2EDuration="2.555372243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:25.554612711 +0000 UTC m=+1372.206111238" watchObservedRunningTime="2026-10-29 23:05:25.555372243 +0000 UTC m=+1372.206870775" Oct 29 23:05:33.403723 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:33.403689 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4" Oct 29 23:05:33.914993 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:05:33.914947 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:05:36.469754 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:36.469721 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv" Oct 29 23:05:36.546670 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:36.546638 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5" Oct 29 23:05:37.483527 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:37.483498 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v" Oct 29 23:05:44.823209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:44.816858 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:05:44.823209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:44.816936 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" podUID="0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:05:46.991722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:46.991689 5057 generic.go:356] "Generic (PLEG): container finished" podID="0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" containerID="e4e8a3003c3c34e7083d941ab9221ee7feefdcae06a8aefa92c99f5038b1a4ff" exitCode=0 Oct 29 23:05:46.992207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:46.991759 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" event={"ID":"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1","Type":"ContainerDied","Data":"e4e8a3003c3c34e7083d941ab9221ee7feefdcae06a8aefa92c99f5038b1a4ff"} Oct 29 23:05:46.992285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:46.992219 5057 scope.go:122] "RemoveContainer" containerID="e4e8a3003c3c34e7083d941ab9221ee7feefdcae06a8aefa92c99f5038b1a4ff" Oct 29 23:05:46.993825 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:46.993787 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:46.993943 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:46.993880 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:46.993943 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:46.993930 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:05:48.002414 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:48.002376 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" event={"ID":"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1","Type":"ContainerStarted","Data":"fe0c5862222696fdebba6329b776e5b27a29da4eb00973d7336bdfd4716f8d08"} Oct 29 23:05:48.825577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:48.825545 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:05:48.825996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:48.825970 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 29 23:05:48.826177 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:48.826151 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" podUID="0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 29 23:05:49.828827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:49.825613 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 29 23:05:49.828827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:49.825692 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" podUID="0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 29 23:05:50.826299 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:50.826255 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 29 23:05:50.826625 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:50.826587 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" podUID="0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 29 23:05:51.845210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:51.844702 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 29 23:05:51.845210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:51.844774 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" podUID="0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 29 23:05:52.846222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:52.846160 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 29 23:05:52.846872 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:52.846826 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" podUID="0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 29 23:05:53.849355 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:53.849322 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:05:53.850996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:53.850957 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:05:53.856442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:53.856410 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:05:56.452592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:56.452558 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp"] Oct 29 23:05:56.457257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:56.457236 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" Oct 29 23:05:56.479015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:56.478980 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp"] Oct 29 23:05:56.481659 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:56.481633 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-dfedfe-serving-cert\"" Oct 29 23:05:56.664915 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:56.645524 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wndxv\" (UniqueName: \"kubernetes.io/projected/3ff64f87-a201-4f46-af24-e1df63c2dab7-kube-api-access-wndxv\") pod \"maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp\" (UID: \"3ff64f87-a201-4f46-af24-e1df63c2dab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" Oct 29 23:05:56.666081 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:56.666048 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ff64f87-a201-4f46-af24-e1df63c2dab7-metrics-certs\") pod \"maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp\" (UID: \"3ff64f87-a201-4f46-af24-e1df63c2dab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" Oct 29 23:05:56.666421 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:56.666382 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff64f87-a201-4f46-af24-e1df63c2dab7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp\" (UID: \"3ff64f87-a201-4f46-af24-e1df63c2dab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" Oct 29 23:05:56.666930 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:56.666904 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ff64f87-a201-4f46-af24-e1df63c2dab7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp\" (UID: \"3ff64f87-a201-4f46-af24-e1df63c2dab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" Oct 29 23:05:56.786473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:56.786381 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff64f87-a201-4f46-af24-e1df63c2dab7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp\" (UID: \"3ff64f87-a201-4f46-af24-e1df63c2dab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" Oct 29 23:05:56.786992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:56.786965 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ff64f87-a201-4f46-af24-e1df63c2dab7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp\" (UID: \"3ff64f87-a201-4f46-af24-e1df63c2dab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" Oct 29 23:05:56.787285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:56.787266 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wndxv\" (UniqueName: \"kubernetes.io/projected/3ff64f87-a201-4f46-af24-e1df63c2dab7-kube-api-access-wndxv\") pod \"maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp\" (UID: \"3ff64f87-a201-4f46-af24-e1df63c2dab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" Oct 29 23:05:56.787499 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:56.787484 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ff64f87-a201-4f46-af24-e1df63c2dab7-metrics-certs\") pod \"maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp\" (UID: \"3ff64f87-a201-4f46-af24-e1df63c2dab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" Oct 29 23:05:56.791888 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:56.791864 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff64f87-a201-4f46-af24-e1df63c2dab7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp\" (UID: \"3ff64f87-a201-4f46-af24-e1df63c2dab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" Oct 29 23:05:56.809339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:56.807657 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ff64f87-a201-4f46-af24-e1df63c2dab7-metrics-certs\") pod \"maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp\" (UID: \"3ff64f87-a201-4f46-af24-e1df63c2dab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" Oct 29 23:05:56.818003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:56.813172 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ff64f87-a201-4f46-af24-e1df63c2dab7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp\" (UID: \"3ff64f87-a201-4f46-af24-e1df63c2dab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" Oct 29 23:05:56.819516 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:56.819493 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wndxv\" (UniqueName: \"kubernetes.io/projected/3ff64f87-a201-4f46-af24-e1df63c2dab7-kube-api-access-wndxv\") pod \"maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp\" (UID: \"3ff64f87-a201-4f46-af24-e1df63c2dab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" Oct 29 23:05:57.073464 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:57.073370 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" Oct 29 23:05:57.437453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:57.437423 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp"] Oct 29 23:05:57.438563 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:05:57.438533 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff64f87_a201_4f46_af24_e1df63c2dab7.slice/crio-b58dcc2c2607b3aa3d83a03e35aa2b211726bf347c1124b033784261ea557ff8 WatchSource:0}: Error finding container b58dcc2c2607b3aa3d83a03e35aa2b211726bf347c1124b033784261ea557ff8: Status 404 returned error can't find the container with id b58dcc2c2607b3aa3d83a03e35aa2b211726bf347c1124b033784261ea557ff8 Oct 29 23:05:58.408320 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:58.408277 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" event={"ID":"3ff64f87-a201-4f46-af24-e1df63c2dab7","Type":"ContainerStarted","Data":"14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40"} Oct 29 23:05:58.408766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:58.408340 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" event={"ID":"3ff64f87-a201-4f46-af24-e1df63c2dab7","Type":"ContainerStarted","Data":"b58dcc2c2607b3aa3d83a03e35aa2b211726bf347c1124b033784261ea557ff8"} Oct 29 23:05:58.408766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:58.408427 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" Oct 29 23:05:58.465248 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:05:58.465103 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" podStartSLOduration=1.979866442 podStartE2EDuration="2.465082035s" totalImagesPullingTime="485.215593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:58.458011772 +0000 UTC m=+1405.109510303" watchObservedRunningTime="2026-10-29 23:05:58.465082035 +0000 UTC m=+1405.116580566" Oct 29 23:06:03.921642 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:06:03.921597 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:06:04.421712 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:04.421683 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" Oct 29 23:06:06.061206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.016363 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh"] Oct 29 23:06:06.282922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.282882 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh"] Oct 29 23:06:06.283241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.283170 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz"] Oct 29 23:06:06.289337 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.289314 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh" Oct 29 23:06:06.329678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.329592 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-41b12c\"" Oct 29 23:06:06.330181 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.330159 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-41b12c-dockercfg-hn4cw\"" Oct 29 23:06:06.399284 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.399253 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz" Oct 29 23:06:06.402930 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.402906 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c"] Oct 29 23:06:06.403394 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.403365 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" podUID="beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" containerName="istio-proxy" containerID="cri-o://e4d59e053cf13523c9948891f7c5f4815f293a943d59f20bf4c93006d69e95a4" gracePeriod=30 Oct 29 23:06:06.420334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.420268 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9d94\" (UniqueName: \"kubernetes.io/projected/55bfce1f-8524-43aa-949d-e933d41ca3cb-kube-api-access-z9d94\") pod \"payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh\" (UID: \"55bfce1f-8524-43aa-949d-e933d41ca3cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh" Oct 29 23:06:06.420575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.420555 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bfce1f-8524-43aa-949d-e933d41ca3cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh\" (UID: \"55bfce1f-8524-43aa-949d-e933d41ca3cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh" Oct 29 23:06:06.468224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.468179 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l"] Oct 29 23:06:06.468853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.468822 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" podUID="196b11f8-6a98-45c1-800b-8de400a902bf" containerName="istio-proxy" containerID="cri-o://a03b2f8f98c2fa185501363e63898766a3f91276265c79d1f983e000d7731aba" gracePeriod=30 Oct 29 23:06:06.475392 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.475363 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz"] Oct 29 23:06:06.522749 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.522712 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3361dd3-0ed7-4b16-aeb4-17633004627a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz\" (UID: \"e3361dd3-0ed7-4b16-aeb4-17633004627a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz" Oct 29 23:06:06.522962 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.522791 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkjrs\" (UniqueName: \"kubernetes.io/projected/e3361dd3-0ed7-4b16-aeb4-17633004627a-kube-api-access-vkjrs\") pod \"payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz\" (UID: \"e3361dd3-0ed7-4b16-aeb4-17633004627a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz" Oct 29 23:06:06.522962 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.522945 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9d94\" (UniqueName: \"kubernetes.io/projected/55bfce1f-8524-43aa-949d-e933d41ca3cb-kube-api-access-z9d94\") pod \"payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh\" (UID: \"55bfce1f-8524-43aa-949d-e933d41ca3cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh" Oct 29 23:06:06.523261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.523227 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bfce1f-8524-43aa-949d-e933d41ca3cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh\" (UID: \"55bfce1f-8524-43aa-949d-e933d41ca3cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh" Oct 29 23:06:06.524178 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.524152 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bfce1f-8524-43aa-949d-e933d41ca3cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh\" (UID: \"55bfce1f-8524-43aa-949d-e933d41ca3cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh" Oct 29 23:06:06.550611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.550577 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9d94\" (UniqueName: \"kubernetes.io/projected/55bfce1f-8524-43aa-949d-e933d41ca3cb-kube-api-access-z9d94\") pod \"payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh\" (UID: \"55bfce1f-8524-43aa-949d-e933d41ca3cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh" Oct 29 23:06:06.628273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.627386 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3361dd3-0ed7-4b16-aeb4-17633004627a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz\" (UID: \"e3361dd3-0ed7-4b16-aeb4-17633004627a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz" Oct 29 23:06:06.628273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.627460 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkjrs\" (UniqueName: \"kubernetes.io/projected/e3361dd3-0ed7-4b16-aeb4-17633004627a-kube-api-access-vkjrs\") pod \"payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz\" (UID: \"e3361dd3-0ed7-4b16-aeb4-17633004627a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz" Oct 29 23:06:06.629221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.629115 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3361dd3-0ed7-4b16-aeb4-17633004627a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz\" (UID: \"e3361dd3-0ed7-4b16-aeb4-17633004627a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz" Oct 29 23:06:06.678289 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.672950 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkjrs\" (UniqueName: \"kubernetes.io/projected/e3361dd3-0ed7-4b16-aeb4-17633004627a-kube-api-access-vkjrs\") pod \"payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz\" (UID: \"e3361dd3-0ed7-4b16-aeb4-17633004627a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz" Oct 29 23:06:06.743792 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.735372 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh" Oct 29 23:06:06.773987 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:06.773951 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz" Oct 29 23:06:07.147674 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:06:07.147638 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55bfce1f_8524_43aa_949d_e933d41ca3cb.slice/crio-fb9d3c758594003d896edfac494c281aaed0cb01a5d3fe0e1810f3d63f919a55 WatchSource:0}: Error finding container fb9d3c758594003d896edfac494c281aaed0cb01a5d3fe0e1810f3d63f919a55: Status 404 returned error can't find the container with id fb9d3c758594003d896edfac494c281aaed0cb01a5d3fe0e1810f3d63f919a55 Oct 29 23:06:07.148124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:07.148101 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh"] Oct 29 23:06:07.159804 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:07.159778 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz"] Oct 29 23:06:07.162766 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:06:07.162733 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3361dd3_0ed7_4b16_aeb4_17633004627a.slice/crio-ff6032ed36dce654b2f5cbb95d63c42b5fbe754b311522bd2e9119a1727ace95 WatchSource:0}: Error finding container ff6032ed36dce654b2f5cbb95d63c42b5fbe754b311522bd2e9119a1727ace95: Status 404 returned error can't find the container with id ff6032ed36dce654b2f5cbb95d63c42b5fbe754b311522bd2e9119a1727ace95 Oct 29 23:06:07.708272 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:07.707984 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh" event={"ID":"55bfce1f-8524-43aa-949d-e933d41ca3cb","Type":"ContainerStarted","Data":"fb9d3c758594003d896edfac494c281aaed0cb01a5d3fe0e1810f3d63f919a55"} Oct 29 23:06:07.708524 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:07.708504 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz" event={"ID":"e3361dd3-0ed7-4b16-aeb4-17633004627a","Type":"ContainerStarted","Data":"ff6032ed36dce654b2f5cbb95d63c42b5fbe754b311522bd2e9119a1727ace95"} Oct 29 23:06:08.710723 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:08.710686 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh" event={"ID":"55bfce1f-8524-43aa-949d-e933d41ca3cb","Type":"ContainerStarted","Data":"a8a724a9f3104ff32e2cceecb47fd6fc7ac629032bb03173cc2e5fb2e2822652"} Oct 29 23:06:08.711230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:08.710757 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh" Oct 29 23:06:08.712716 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:08.712689 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz" event={"ID":"e3361dd3-0ed7-4b16-aeb4-17633004627a","Type":"ContainerStarted","Data":"368d495962b7d090507df19ab247f779bdc3f485d8525f7277aa715f4c4cba37"} Oct 29 23:06:08.712857 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:08.712758 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz" Oct 29 23:06:08.760170 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:08.760103 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh" podStartSLOduration=3.760084734 podStartE2EDuration="3.760084734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:08.757413216 +0000 UTC m=+1415.408911747" watchObservedRunningTime="2026-10-29 23:06:08.760084734 +0000 UTC m=+1415.411583269" Oct 29 23:06:08.805256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:08.805167 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz" podStartSLOduration=3.805148296 podStartE2EDuration="3.805148296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:08.801842238 +0000 UTC m=+1415.453340771" watchObservedRunningTime="2026-10-29 23:06:08.805148296 +0000 UTC m=+1415.456646828" Oct 29 23:06:11.744521 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.744445 5057 generic.go:356] "Generic (PLEG): container finished" podID="196b11f8-6a98-45c1-800b-8de400a902bf" containerID="a03b2f8f98c2fa185501363e63898766a3f91276265c79d1f983e000d7731aba" exitCode=0 Oct 29 23:06:11.744972 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.744944 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" event={"ID":"196b11f8-6a98-45c1-800b-8de400a902bf","Type":"ContainerDied","Data":"a03b2f8f98c2fa185501363e63898766a3f91276265c79d1f983e000d7731aba"} Oct 29 23:06:11.746897 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.746869 5057 generic.go:356] "Generic (PLEG): container finished" podID="beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" containerID="e4d59e053cf13523c9948891f7c5f4815f293a943d59f20bf4c93006d69e95a4" exitCode=0 Oct 29 23:06:11.747011 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.746907 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" event={"ID":"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76","Type":"ContainerDied","Data":"e4d59e053cf13523c9948891f7c5f4815f293a943d59f20bf4c93006d69e95a4"} Oct 29 23:06:11.795520 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.795491 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 23:06:11.904837 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.904789 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-workload-socket\") pod \"196b11f8-6a98-45c1-800b-8de400a902bf\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " Oct 29 23:06:11.905069 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.904905 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-workload-certs\") pod \"196b11f8-6a98-45c1-800b-8de400a902bf\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " Oct 29 23:06:11.905069 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.904996 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-istio-envoy\") pod \"196b11f8-6a98-45c1-800b-8de400a902bf\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " Oct 29 23:06:11.905252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.905092 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/196b11f8-6a98-45c1-800b-8de400a902bf-kube-api-access-kkq6f\" (UniqueName: \"kubernetes.io/projected/196b11f8-6a98-45c1-800b-8de400a902bf-kube-api-access-kkq6f\") pod \"196b11f8-6a98-45c1-800b-8de400a902bf\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " Oct 29 23:06:11.905252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.905151 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/196b11f8-6a98-45c1-800b-8de400a902bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196b11f8-6a98-45c1-800b-8de400a902bf-istiod-ca-cert\") pod \"196b11f8-6a98-45c1-800b-8de400a902bf\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " Oct 29 23:06:11.905252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.905244 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/196b11f8-6a98-45c1-800b-8de400a902bf-istio-token\" (UniqueName: \"kubernetes.io/projected/196b11f8-6a98-45c1-800b-8de400a902bf-istio-token\") pod \"196b11f8-6a98-45c1-800b-8de400a902bf\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " Oct 29 23:06:11.905438 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.905282 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-istio-data\") pod \"196b11f8-6a98-45c1-800b-8de400a902bf\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " Oct 29 23:06:11.905494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.905477 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/196b11f8-6a98-45c1-800b-8de400a902bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196b11f8-6a98-45c1-800b-8de400a902bf-istio-podinfo\") pod \"196b11f8-6a98-45c1-800b-8de400a902bf\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " Oct 29 23:06:11.905550 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.905509 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-credential-socket\") pod \"196b11f8-6a98-45c1-800b-8de400a902bf\" (UID: \"196b11f8-6a98-45c1-800b-8de400a902bf\") " Oct 29 23:06:11.905938 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.905911 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-workload-certs" pod "196b11f8-6a98-45c1-800b-8de400a902bf" (UID: "196b11f8-6a98-45c1-800b-8de400a902bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:11.906489 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.906278 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-workload-socket" pod "196b11f8-6a98-45c1-800b-8de400a902bf" (UID: "196b11f8-6a98-45c1-800b-8de400a902bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:11.906602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.906498 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:06:11.906602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.906519 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:06:11.906981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.906958 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/196b11f8-6a98-45c1-800b-8de400a902bf-istiod-ca-cert" pod "196b11f8-6a98-45c1-800b-8de400a902bf" (UID: "196b11f8-6a98-45c1-800b-8de400a902bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:11.907427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.907393 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-istio-data" pod "196b11f8-6a98-45c1-800b-8de400a902bf" (UID: "196b11f8-6a98-45c1-800b-8de400a902bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:11.908452 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.907773 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-credential-socket" pod "196b11f8-6a98-45c1-800b-8de400a902bf" (UID: "196b11f8-6a98-45c1-800b-8de400a902bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:11.910510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.910484 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196b11f8-6a98-45c1-800b-8de400a902bf-istio-token" pod "196b11f8-6a98-45c1-800b-8de400a902bf" (UID: "196b11f8-6a98-45c1-800b-8de400a902bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:11.911923 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.911707 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196b11f8-6a98-45c1-800b-8de400a902bf-kube-api-access-kkq6f" pod "196b11f8-6a98-45c1-800b-8de400a902bf" (UID: "196b11f8-6a98-45c1-800b-8de400a902bf"). InnerVolumeSpecName "kube-api-access-kkq6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:11.911923 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.911871 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/196b11f8-6a98-45c1-800b-8de400a902bf-istio-podinfo" pod "196b11f8-6a98-45c1-800b-8de400a902bf" (UID: "196b11f8-6a98-45c1-800b-8de400a902bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:11.919392 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:11.919359 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-istio-envoy" pod "196b11f8-6a98-45c1-800b-8de400a902bf" (UID: "196b11f8-6a98-45c1-800b-8de400a902bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:12.008234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.008138 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196b11f8-6a98-45c1-800b-8de400a902bf-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.008234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.008169 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196b11f8-6a98-45c1-800b-8de400a902bf-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.008234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.008178 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.008234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.008200 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196b11f8-6a98-45c1-800b-8de400a902bf-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.008234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.008208 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.008234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.008217 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196b11f8-6a98-45c1-800b-8de400a902bf-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.008234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.008228 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkq6f\" (UniqueName: \"kubernetes.io/projected/196b11f8-6a98-45c1-800b-8de400a902bf-kube-api-access-kkq6f\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.083785 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.083749 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7"] Oct 29 23:06:12.084618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.084602 5057 cpu_manager.go:414] "containerMap: removing container" podUID="196b11f8-6a98-45c1-800b-8de400a902bf" containerName="istio-proxy" Oct 29 23:06:12.084698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.084618 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="196b11f8-6a98-45c1-800b-8de400a902bf" containerName="istio-proxy" Oct 29 23:06:12.084808 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.084797 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="196b11f8-6a98-45c1-800b-8de400a902bf" containerName="istio-proxy" Oct 29 23:06:12.088564 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.088539 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7" Oct 29 23:06:12.114655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.112818 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-957eef-dockercfg-dlc7b\"" Oct 29 23:06:12.120787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.120755 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7"] Oct 29 23:06:12.241211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.240405 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-957eef\"" Oct 29 23:06:12.309277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.302898 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7765cde8-749b-4360-858b-b56100c5037d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7\" (UID: \"7765cde8-749b-4360-858b-b56100c5037d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7" Oct 29 23:06:12.309277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.303119 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sghzx\" (UniqueName: \"kubernetes.io/projected/7765cde8-749b-4360-858b-b56100c5037d-kube-api-access-sghzx\") pod \"payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7\" (UID: \"7765cde8-749b-4360-858b-b56100c5037d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7" Oct 29 23:06:12.452568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.436483 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7765cde8-749b-4360-858b-b56100c5037d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7\" (UID: \"7765cde8-749b-4360-858b-b56100c5037d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7" Oct 29 23:06:12.452568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.436751 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sghzx\" (UniqueName: \"kubernetes.io/projected/7765cde8-749b-4360-858b-b56100c5037d-kube-api-access-sghzx\") pod \"payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7\" (UID: \"7765cde8-749b-4360-858b-b56100c5037d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7" Oct 29 23:06:12.452568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.438048 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7765cde8-749b-4360-858b-b56100c5037d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7\" (UID: \"7765cde8-749b-4360-858b-b56100c5037d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7" Oct 29 23:06:12.549208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.547930 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sghzx\" (UniqueName: \"kubernetes.io/projected/7765cde8-749b-4360-858b-b56100c5037d-kube-api-access-sghzx\") pod \"payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7\" (UID: \"7765cde8-749b-4360-858b-b56100c5037d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7" Oct 29 23:06:12.620937 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.620908 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 23:06:12.647262 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.647231 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-envoy\") pod \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " Oct 29 23:06:12.647442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.647324 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-kube-api-access-njzrc\" (UniqueName: \"kubernetes.io/projected/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-kube-api-access-njzrc\") pod \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " Oct 29 23:06:12.647442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.647436 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-data\") pod \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " Oct 29 23:06:12.647573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.647475 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-token\" (UniqueName: \"kubernetes.io/projected/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-token\") pod \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " Oct 29 23:06:12.647573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.647502 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-podinfo\") pod \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " Oct 29 23:06:12.647693 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.647607 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istiod-ca-cert\") pod \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " Oct 29 23:06:12.647792 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.647767 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-data" pod "beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" (UID: "beb05fe4-ceb9-4a1c-8042-2a2b656eaa76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:12.647954 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.647932 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-workload-certs\") pod \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " Oct 29 23:06:12.648034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.648016 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-workload-socket\") pod \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " Oct 29 23:06:12.648107 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.648085 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-credential-socket\") pod \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\" (UID: \"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76\") " Oct 29 23:06:12.648166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.648125 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istiod-ca-cert" pod "beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" (UID: "beb05fe4-ceb9-4a1c-8042-2a2b656eaa76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:12.648515 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.648309 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-workload-socket" pod "beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" (UID: "beb05fe4-ceb9-4a1c-8042-2a2b656eaa76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:12.648515 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.648451 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-credential-socket" pod "beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" (UID: "beb05fe4-ceb9-4a1c-8042-2a2b656eaa76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:12.648515 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.648474 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-workload-certs" pod "beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" (UID: "beb05fe4-ceb9-4a1c-8042-2a2b656eaa76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:12.649145 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.649120 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.649324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.649151 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.649324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.649165 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.649324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.649179 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.649324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.649208 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.649807 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.649760 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-envoy" pod "beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" (UID: "beb05fe4-ceb9-4a1c-8042-2a2b656eaa76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:12.650100 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.650080 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-token" pod "beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" (UID: "beb05fe4-ceb9-4a1c-8042-2a2b656eaa76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:12.650687 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.650657 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-podinfo" pod "beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" (UID: "beb05fe4-ceb9-4a1c-8042-2a2b656eaa76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:12.651260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.651237 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-kube-api-access-njzrc" pod "beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" (UID: "beb05fe4-ceb9-4a1c-8042-2a2b656eaa76"). InnerVolumeSpecName "kube-api-access-njzrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:12.722219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.722137 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7" Oct 29 23:06:12.751499 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.751465 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.751499 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.751501 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.752009 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.751512 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njzrc\" (UniqueName: \"kubernetes.io/projected/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-kube-api-access-njzrc\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.752009 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.751527 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:06:12.760581 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.760554 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" Oct 29 23:06:12.761627 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.761587 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l" event={"ID":"196b11f8-6a98-45c1-800b-8de400a902bf","Type":"ContainerDied","Data":"35632aece0edf259d373b54c1f0a2c932a7ac65c639c55a4af42d4c9bc91bf04"} Oct 29 23:06:12.761777 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.761681 5057 scope.go:122] "RemoveContainer" containerID="a03b2f8f98c2fa185501363e63898766a3f91276265c79d1f983e000d7731aba" Oct 29 23:06:12.765953 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.763469 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" Oct 29 23:06:12.765953 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.763524 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c" event={"ID":"beb05fe4-ceb9-4a1c-8042-2a2b656eaa76","Type":"ContainerDied","Data":"21f9f6d327a9fa53c70df54bbe81e81e56c88122c49ded4a5b0d9c7f9ac170a2"} Oct 29 23:06:12.807711 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.791422 5057 scope.go:122] "RemoveContainer" containerID="e4d59e053cf13523c9948891f7c5f4815f293a943d59f20bf4c93006d69e95a4" Oct 29 23:06:12.834757 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.829448 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l"] Oct 29 23:06:12.836427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.836366 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-nlz8l"] Oct 29 23:06:12.877869 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.877840 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c"] Oct 29 23:06:12.887281 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:12.887252 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-xbn5c"] Oct 29 23:06:13.341214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:13.335572 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7"] Oct 29 23:06:13.493018 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:13.492985 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="196b11f8-6a98-45c1-800b-8de400a902bf" path="/var/lib/kubelet/pods/196b11f8-6a98-45c1-800b-8de400a902bf/volumes" Oct 29 23:06:13.494030 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:13.494009 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" path="/var/lib/kubelet/pods/beb05fe4-ceb9-4a1c-8042-2a2b656eaa76/volumes" Oct 29 23:06:13.775458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:13.775359 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7" event={"ID":"7765cde8-749b-4360-858b-b56100c5037d","Type":"ContainerStarted","Data":"854d30196b12d63b94d373e2980945d78281dc70c35a72e41886af8c2db0a7a9"} Oct 29 23:06:13.775458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:13.775429 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7" event={"ID":"7765cde8-749b-4360-858b-b56100c5037d","Type":"ContainerStarted","Data":"230e185f7baaac7c9e50bbb9552df9fea9d8510c6686c2bf702da3b60b8fa03b"} Oct 29 23:06:13.775458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:13.775456 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7" Oct 29 23:06:13.824345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:13.824292 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7" podStartSLOduration=1.824276387 podStartE2EDuration="1.824276387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:13.819208127 +0000 UTC m=+1420.470706648" watchObservedRunningTime="2026-10-29 23:06:13.824276387 +0000 UTC m=+1420.475774917" Oct 29 23:06:14.654158 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:14.654122 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:06:19.749212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:19.743356 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz" Oct 29 23:06:19.749853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:19.749829 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh" Oct 29 23:06:24.892930 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:24.892895 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7" Oct 29 23:06:26.904751 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:26.904714 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh_55bfce1f-8524-43aa-949d-e933d41ca3cb/payload-processing/0.log" Oct 29 23:06:28.447519 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:28.447484 5057 generic.go:356] "Generic (PLEG): container finished" podID="fdd750a6-8487-4d03-9e52-4c6b6ef9133b" containerID="d88e4d7855571e464b282f6e02800d1afca39961e0b9350fcff4a656bc6a40c4" exitCode=0 Oct 29 23:06:28.448008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:28.447561 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" event={"ID":"fdd750a6-8487-4d03-9e52-4c6b6ef9133b","Type":"ContainerDied","Data":"d88e4d7855571e464b282f6e02800d1afca39961e0b9350fcff4a656bc6a40c4"} Oct 29 23:06:28.448107 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:28.448089 5057 scope.go:122] "RemoveContainer" containerID="d88e4d7855571e464b282f6e02800d1afca39961e0b9350fcff4a656bc6a40c4" Oct 29 23:06:28.450060 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:28.450028 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:06:28.450201 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:28.450110 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:06:28.450201 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:28.450162 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:06:29.489067 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:29.489030 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" event={"ID":"fdd750a6-8487-4d03-9e52-4c6b6ef9133b","Type":"ContainerStarted","Data":"6cd18c7f34df3865986b41baca418b6ff2c4ea914420ff8a0112a95789549aa1"} Oct 29 23:06:30.387967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:30.387934 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:06:30.388143 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:30.388098 5057 patch_prober.go:29] interesting pod/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 29 23:06:30.388227 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:30.388169 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" podUID="fdd750a6-8487-4d03-9e52-4c6b6ef9133b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 29 23:06:30.490128 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:30.490081 5057 generic.go:356] "Generic (PLEG): container finished" podID="0cb218f5-5d38-4afb-9357-0b2db3da958e" containerID="73a3f54198ffc6a3e5f5d9850cb7f9890874877f74fee6af44d39047b93ae598" exitCode=0 Oct 29 23:06:30.490673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:30.490176 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" event={"ID":"0cb218f5-5d38-4afb-9357-0b2db3da958e","Type":"ContainerDied","Data":"73a3f54198ffc6a3e5f5d9850cb7f9890874877f74fee6af44d39047b93ae598"} Oct 29 23:06:30.490861 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:30.490834 5057 scope.go:122] "RemoveContainer" containerID="73a3f54198ffc6a3e5f5d9850cb7f9890874877f74fee6af44d39047b93ae598" Oct 29 23:06:30.492321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:30.492299 5057 generic.go:356] "Generic (PLEG): container finished" podID="06c26371-4cdf-4bf1-95d7-855d452655c3" containerID="95de34c0859f5b7e5cbf48288a72e2d852ca439d02b673d5b3619997dec7168e" exitCode=0 Oct 29 23:06:30.492421 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:30.492373 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" event={"ID":"06c26371-4cdf-4bf1-95d7-855d452655c3","Type":"ContainerDied","Data":"95de34c0859f5b7e5cbf48288a72e2d852ca439d02b673d5b3619997dec7168e"} Oct 29 23:06:30.492950 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:30.492919 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:06:30.492993 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:30.492974 5057 scope.go:122] "RemoveContainer" containerID="95de34c0859f5b7e5cbf48288a72e2d852ca439d02b673d5b3619997dec7168e" Oct 29 23:06:30.493027 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:30.493007 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:06:30.493075 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:30.493059 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:06:30.494652 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:30.494625 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:06:30.494762 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:30.494689 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:06:30.494762 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:30.494725 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:06:31.388099 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:31.388056 5057 patch_prober.go:29] interesting pod/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 29 23:06:31.388318 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:31.388114 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" podUID="fdd750a6-8487-4d03-9e52-4c6b6ef9133b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 29 23:06:31.502351 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:31.502316 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" event={"ID":"0cb218f5-5d38-4afb-9357-0b2db3da958e","Type":"ContainerStarted","Data":"c899d69ba8ce6ec5b8a5f76c066cf08750c7eff19b0d0050d6ccdfa8190931e5"} Oct 29 23:06:31.504590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:31.504565 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" event={"ID":"06c26371-4cdf-4bf1-95d7-855d452655c3","Type":"ContainerStarted","Data":"c5745238b5490434784905f117394c19ec0f52ac9c64389581fb85bf55fb72fd"} Oct 29 23:06:31.710240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:31.710159 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:06:31.715367 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:31.715330 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 29 23:06:31.715666 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:31.715629 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" podUID="06c26371-4cdf-4bf1-95d7-855d452655c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 29 23:06:31.773358 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:31.773324 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:06:31.796600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:31.796563 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:06:31.804387 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:31.804344 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 29 23:06:31.804672 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:31.804639 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" podUID="0cb218f5-5d38-4afb-9357-0b2db3da958e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 29 23:06:31.841411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:31.841304 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:06:32.389681 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:32.389620 5057 patch_prober.go:29] interesting pod/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 29 23:06:32.391055 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:32.391022 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" podUID="fdd750a6-8487-4d03-9e52-4c6b6ef9133b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 29 23:06:32.720265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:32.710677 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 29 23:06:32.720265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:32.710735 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" podUID="06c26371-4cdf-4bf1-95d7-855d452655c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 29 23:06:32.805163 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:32.802440 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 29 23:06:32.805163 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:32.802499 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" podUID="0cb218f5-5d38-4afb-9357-0b2db3da958e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 29 23:06:33.400212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:33.399320 5057 patch_prober.go:29] interesting pod/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 29 23:06:33.400212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:33.399385 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" podUID="fdd750a6-8487-4d03-9e52-4c6b6ef9133b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 29 23:06:33.734267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:33.723629 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 29 23:06:33.734267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:33.723686 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" podUID="06c26371-4cdf-4bf1-95d7-855d452655c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 29 23:06:33.819372 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:33.819325 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 29 23:06:33.819698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:33.819664 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" podUID="0cb218f5-5d38-4afb-9357-0b2db3da958e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 29 23:06:34.286472 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:06:34.286429 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:06:34.533310 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:34.510425 5057 patch_prober.go:29] interesting pod/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 29 23:06:34.533310 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:34.510484 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" podUID="fdd750a6-8487-4d03-9e52-4c6b6ef9133b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 29 23:06:34.604295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:34.581655 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:06:34.613979 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:34.613285 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:06:34.665212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:34.651436 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 23:06:34.873296 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:34.782448 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:06:34.887229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:34.877681 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 29 23:06:34.887229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:34.877736 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" podUID="06c26371-4cdf-4bf1-95d7-855d452655c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 29 23:06:34.887229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:34.882442 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 29 23:06:34.887229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:34.882488 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" podUID="0cb218f5-5d38-4afb-9357-0b2db3da958e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 29 23:06:35.429237 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:35.422945 5057 patch_prober.go:29] interesting pod/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 29 23:06:35.429237 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:35.423025 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" podUID="fdd750a6-8487-4d03-9e52-4c6b6ef9133b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 29 23:06:35.711569 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:35.711475 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 29 23:06:35.711805 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:35.711779 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" podUID="06c26371-4cdf-4bf1-95d7-855d452655c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 29 23:06:35.800426 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:35.800359 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 29 23:06:35.800760 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:35.800703 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" podUID="0cb218f5-5d38-4afb-9357-0b2db3da958e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 29 23:06:36.356265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:36.355768 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:06:36.393163 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:36.393131 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:06:36.707470 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:36.707438 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 29 23:06:36.707625 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:36.707486 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" podUID="06c26371-4cdf-4bf1-95d7-855d452655c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 29 23:06:36.797752 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:36.797711 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 29 23:06:36.797963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:36.797762 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" podUID="0cb218f5-5d38-4afb-9357-0b2db3da958e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 29 23:06:37.214306 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:37.214277 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:06:37.706596 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:37.706553 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 29 23:06:37.707166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:37.706617 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" podUID="06c26371-4cdf-4bf1-95d7-855d452655c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 29 23:06:37.797285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:37.797247 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 29 23:06:37.797430 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:37.797353 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" podUID="0cb218f5-5d38-4afb-9357-0b2db3da958e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 29 23:06:38.711759 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:38.711728 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:06:38.712941 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:38.712922 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:06:38.802898 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:38.802865 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:06:38.805630 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:38.805605 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:06:41.885686 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:41.885639 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52"] Oct 29 23:06:41.887667 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:41.887643 5057 cpu_manager.go:414] "containerMap: removing container" podUID="beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" containerName="istio-proxy" Oct 29 23:06:41.887862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:41.887833 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" containerName="istio-proxy" Oct 29 23:06:41.888273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:41.888253 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="beb05fe4-ceb9-4a1c-8042-2a2b656eaa76" containerName="istio-proxy" Oct 29 23:06:41.895114 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:41.895093 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:41.896842 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:41.896814 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52"] Oct 29 23:06:41.917031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:41.916508 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-cd000a-openshift-default-dockercfg-gc9bl\"" Oct 29 23:06:42.050309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.045477 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-credential-socket\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.050309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.045621 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-podinfo\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.050309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.045747 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-workload-socket\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.050309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.045775 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-token\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.050309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.045833 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istiod-ca-cert\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.050309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.045864 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-data\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.050309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.045943 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-workload-certs\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.050309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.046013 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8clv\" (UniqueName: \"kubernetes.io/projected/8fd55749-31a3-49ff-8965-057fb2eeb9b8-kube-api-access-b8clv\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.050309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.046109 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-envoy\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.190771 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.190734 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-podinfo\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.191685 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.191656 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-workload-socket\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.192289 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.192269 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-token\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.213357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.213325 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istiod-ca-cert\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.221226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.214558 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-data\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.221922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.192222 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-workload-socket\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.229910 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.221777 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-data\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.259212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.214310 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istiod-ca-cert\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.261848 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.222160 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-workload-certs\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.261848 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.260341 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8clv\" (UniqueName: \"kubernetes.io/projected/8fd55749-31a3-49ff-8965-057fb2eeb9b8-kube-api-access-b8clv\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.261848 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.260464 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-envoy\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.261848 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.260552 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-credential-socket\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.261848 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.261137 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-credential-socket\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.261848 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.222454 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-workload-certs\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.502279 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.486356 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-podinfo\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.506599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.506497 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8clv\" (UniqueName: \"kubernetes.io/projected/8fd55749-31a3-49ff-8965-057fb2eeb9b8-kube-api-access-b8clv\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.571210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.546696 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-envoy\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.572286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.551015 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-token\") pod \"e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:42.782269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:42.754891 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:43.457784 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:43.457756 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52"] Oct 29 23:06:43.460638 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:06:43.460608 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd55749_31a3_49ff_8965_057fb2eeb9b8.slice/crio-9edc1ac63d204430079de1878674a43edd121905a33bab5bfdec2aafc345a545 WatchSource:0}: Error finding container 9edc1ac63d204430079de1878674a43edd121905a33bab5bfdec2aafc345a545: Status 404 returned error can't find the container with id 9edc1ac63d204430079de1878674a43edd121905a33bab5bfdec2aafc345a545 Oct 29 23:06:43.463608 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:43.463575 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:06:43.463710 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:43.463639 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:06:43.463710 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:43.463670 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:06:43.983447 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:43.983391 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk"] Oct 29 23:06:43.994207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:43.994170 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:43.995353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:43.995333 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk"] Oct 29 23:06:44.015570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.015532 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-istio-envoy\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.015724 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.015657 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a475a5-9dd5-4278-9109-97682e026b7c-istio-podinfo\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.015802 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.015745 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-istio-data\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.015864 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.015829 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-workload-socket\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.015923 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.015861 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d68rh\" (UniqueName: \"kubernetes.io/projected/83a475a5-9dd5-4278-9109-97682e026b7c-kube-api-access-d68rh\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.015923 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.015890 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a475a5-9dd5-4278-9109-97682e026b7c-istio-token\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.016039 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.015997 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-workload-certs\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.016118 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.016099 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a475a5-9dd5-4278-9109-97682e026b7c-istiod-ca-cert\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.016179 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.016159 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-credential-socket\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.088567 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.088530 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-c23236-openshift-default-dockercfg-rqn9l\"" Oct 29 23:06:44.228220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.148669 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a475a5-9dd5-4278-9109-97682e026b7c-istio-podinfo\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.228220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.148793 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-istio-data\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.228220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.148885 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-workload-socket\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.228220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.148913 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d68rh\" (UniqueName: \"kubernetes.io/projected/83a475a5-9dd5-4278-9109-97682e026b7c-kube-api-access-d68rh\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.228220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.148939 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a475a5-9dd5-4278-9109-97682e026b7c-istio-token\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.228220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.149044 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-workload-certs\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.228220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.149148 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a475a5-9dd5-4278-9109-97682e026b7c-istiod-ca-cert\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.228220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.149199 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-credential-socket\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.228220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.149301 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-istio-envoy\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.228220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.198934 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a475a5-9dd5-4278-9109-97682e026b7c-istio-podinfo\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.228220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.199223 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-istio-data\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.228220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.199475 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-workload-socket\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.261992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.261914 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-workload-certs\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.275207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.264616 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a475a5-9dd5-4278-9109-97682e026b7c-istiod-ca-cert\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.318222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.278640 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-credential-socket\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.328217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.325842 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d68rh\" (UniqueName: \"kubernetes.io/projected/83a475a5-9dd5-4278-9109-97682e026b7c-kube-api-access-d68rh\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.337892 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.336604 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-istio-envoy\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.401239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.391289 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a475a5-9dd5-4278-9109-97682e026b7c-istio-token\") pod \"e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.595259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.590458 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" event={"ID":"8fd55749-31a3-49ff-8965-057fb2eeb9b8","Type":"ContainerStarted","Data":"7a3ed1f921a2cf7022e03834d6e2feba3cf62997b2a11a5f7c073025df0c75c7"} Oct 29 23:06:44.595259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.590531 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" event={"ID":"8fd55749-31a3-49ff-8965-057fb2eeb9b8","Type":"ContainerStarted","Data":"9edc1ac63d204430079de1878674a43edd121905a33bab5bfdec2aafc345a545"} Oct 29 23:06:44.648316 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.630663 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:44.702858 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.702763 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" podStartSLOduration=3.702729253 podStartE2EDuration="3.702729253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:44.649370821 +0000 UTC m=+1451.300869354" watchObservedRunningTime="2026-10-29 23:06:44.702729253 +0000 UTC m=+1451.354227785" Oct 29 23:06:44.819206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.817566 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:44.886294 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:44.880895 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:45.259383 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:45.259344 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk"] Oct 29 23:06:45.261441 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:06:45.261401 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a475a5_9dd5_4278_9109_97682e026b7c.slice/crio-48e9a3918427694787bc9eb10e88af828cbad7ea68c43fc2a2ab00ce9ad03b68 WatchSource:0}: Error finding container 48e9a3918427694787bc9eb10e88af828cbad7ea68c43fc2a2ab00ce9ad03b68: Status 404 returned error can't find the container with id 48e9a3918427694787bc9eb10e88af828cbad7ea68c43fc2a2ab00ce9ad03b68 Oct 29 23:06:45.264790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:45.264754 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:06:45.264862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:45.264833 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:06:45.264901 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:45.264865 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:06:45.678265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:45.676806 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" event={"ID":"83a475a5-9dd5-4278-9109-97682e026b7c","Type":"ContainerStarted","Data":"02c4745abefb7e2bb3424c728f63190c38ab9a773651bc308637f4ed6bc02be1"} Oct 29 23:06:45.678265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:45.676875 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" event={"ID":"83a475a5-9dd5-4278-9109-97682e026b7c","Type":"ContainerStarted","Data":"48e9a3918427694787bc9eb10e88af828cbad7ea68c43fc2a2ab00ce9ad03b68"} Oct 29 23:06:45.678265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:45.676904 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:45.709237 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:45.699993 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:06:45.762469 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:45.762410 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" podStartSLOduration=2.7471667589999997 podStartE2EDuration="2.747166759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:45.746574226 +0000 UTC m=+1452.398072757" watchObservedRunningTime="2026-10-29 23:06:45.747166759 +0000 UTC m=+1452.398665291" Oct 29 23:06:46.652893 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:46.652855 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:46.748930 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:46.748888 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:47.772498 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:47.772467 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:47.774418 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:47.774083 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:06:49.890613 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:49.890571 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:06:49.891116 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:49.890664 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" podUID="bec61ff1-da16-48fd-97cc-7a6cc85213e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:06:52.004353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:52.004316 5057 generic.go:356] "Generic (PLEG): container finished" podID="bec61ff1-da16-48fd-97cc-7a6cc85213e3" containerID="d97a49bf83f1840cf074f16f1e08c6855c6c11f810cc19211ea2be9eb73e3334" exitCode=0 Oct 29 23:06:52.004786 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:52.004386 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" event={"ID":"bec61ff1-da16-48fd-97cc-7a6cc85213e3","Type":"ContainerDied","Data":"d97a49bf83f1840cf074f16f1e08c6855c6c11f810cc19211ea2be9eb73e3334"} Oct 29 23:06:52.004854 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:52.004839 5057 scope.go:122] "RemoveContainer" containerID="d97a49bf83f1840cf074f16f1e08c6855c6c11f810cc19211ea2be9eb73e3334" Oct 29 23:06:52.006637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:52.006608 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:06:52.006717 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:52.006676 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:06:52.006717 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:52.006708 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:06:53.016228 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:53.016163 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" event={"ID":"bec61ff1-da16-48fd-97cc-7a6cc85213e3","Type":"ContainerStarted","Data":"3f9b2c0b3fe287a05e2ca2c7a4bc65cdac612dd759b006120c91a0ac38202177"} Oct 29 23:06:53.774874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:53.774838 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 29 23:06:53.776708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:53.776672 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9"] Oct 29 23:06:53.777251 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:53.777226 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:06:53.777932 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:53.775231 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" podUID="bec61ff1-da16-48fd-97cc-7a6cc85213e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 29 23:06:53.778381 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:53.778326 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" podUID="84648cac-9c75-47a4-a3d6-df79e46b5fd9" containerName="istio-proxy" containerID="cri-o://5ddb2149716a2f78db3992a65ab20f55e0e78c90ed3af04eefa7b7d6f6c9c388" gracePeriod=30 Oct 29 23:06:54.711212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:54.709780 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 29 23:06:54.711212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:54.709869 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" podUID="bec61ff1-da16-48fd-97cc-7a6cc85213e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 29 23:06:55.687302 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:55.687270 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 29 23:06:55.687482 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:55.687335 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" podUID="bec61ff1-da16-48fd-97cc-7a6cc85213e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 29 23:06:56.429209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:56.425818 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd"] Oct 29 23:06:56.477209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:56.437525 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd"] Oct 29 23:06:56.477209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:56.437646 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd" Oct 29 23:06:56.477209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:56.462058 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-618d6c\"" Oct 29 23:06:56.551212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:56.551103 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5272969-d30b-4e39-93b3-587d000ef056-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd\" (UID: \"e5272969-d30b-4e39-93b3-587d000ef056\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd" Oct 29 23:06:56.551411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:56.551270 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pb2p\" (UniqueName: \"kubernetes.io/projected/e5272969-d30b-4e39-93b3-587d000ef056-kube-api-access-6pb2p\") pod \"payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd\" (UID: \"e5272969-d30b-4e39-93b3-587d000ef056\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd" Oct 29 23:06:56.665636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:56.665591 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5272969-d30b-4e39-93b3-587d000ef056-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd\" (UID: \"e5272969-d30b-4e39-93b3-587d000ef056\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd" Oct 29 23:06:56.665975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:56.665955 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pb2p\" (UniqueName: \"kubernetes.io/projected/e5272969-d30b-4e39-93b3-587d000ef056-kube-api-access-6pb2p\") pod \"payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd\" (UID: \"e5272969-d30b-4e39-93b3-587d000ef056\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd" Oct 29 23:06:56.747272 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:56.732340 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pb2p\" (UniqueName: \"kubernetes.io/projected/e5272969-d30b-4e39-93b3-587d000ef056-kube-api-access-6pb2p\") pod \"payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd\" (UID: \"e5272969-d30b-4e39-93b3-587d000ef056\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd" Oct 29 23:06:56.749260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:56.747996 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5272969-d30b-4e39-93b3-587d000ef056-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd\" (UID: \"e5272969-d30b-4e39-93b3-587d000ef056\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd" Oct 29 23:06:56.806224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:56.798745 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 29 23:06:56.806224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:56.798837 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" podUID="bec61ff1-da16-48fd-97cc-7a6cc85213e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 29 23:06:56.938223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:56.929773 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd" Oct 29 23:06:57.009840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:56.990329 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl"] Oct 29 23:06:57.025388 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:57.016483 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl" Oct 29 23:06:57.025388 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:57.024930 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl"] Oct 29 23:06:57.037394 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:57.035218 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-618d6c-dockercfg-bnlx8\"" Oct 29 23:06:57.113784 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:57.113754 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b843c0e-5b58-46ef-b5cd-712b4acad831-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl\" (UID: \"6b843c0e-5b58-46ef-b5cd-712b4acad831\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl" Oct 29 23:06:57.114120 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:57.114094 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh6wn\" (UniqueName: \"kubernetes.io/projected/6b843c0e-5b58-46ef-b5cd-712b4acad831-kube-api-access-fh6wn\") pod \"payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl\" (UID: \"6b843c0e-5b58-46ef-b5cd-712b4acad831\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl" Oct 29 23:06:57.216696 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:57.215521 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b843c0e-5b58-46ef-b5cd-712b4acad831-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl\" (UID: \"6b843c0e-5b58-46ef-b5cd-712b4acad831\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl" Oct 29 23:06:57.216696 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:57.215634 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh6wn\" (UniqueName: \"kubernetes.io/projected/6b843c0e-5b58-46ef-b5cd-712b4acad831-kube-api-access-fh6wn\") pod \"payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl\" (UID: \"6b843c0e-5b58-46ef-b5cd-712b4acad831\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl" Oct 29 23:06:57.216696 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:57.216383 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b843c0e-5b58-46ef-b5cd-712b4acad831-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl\" (UID: \"6b843c0e-5b58-46ef-b5cd-712b4acad831\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl" Oct 29 23:06:57.262111 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:57.261935 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh6wn\" (UniqueName: \"kubernetes.io/projected/6b843c0e-5b58-46ef-b5cd-712b4acad831-kube-api-access-fh6wn\") pod \"payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl\" (UID: \"6b843c0e-5b58-46ef-b5cd-712b4acad831\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl" Oct 29 23:06:57.372425 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:57.372387 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd"] Oct 29 23:06:57.402955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:57.402920 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl" Oct 29 23:06:57.712261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:57.690057 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 29 23:06:57.712873 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:57.712830 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" podUID="bec61ff1-da16-48fd-97cc-7a6cc85213e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 29 23:06:57.807053 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:57.807013 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl"] Oct 29 23:06:58.410090 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:58.408776 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd" event={"ID":"e5272969-d30b-4e39-93b3-587d000ef056","Type":"ContainerStarted","Data":"04ce560694505d4f88ad0c756977e53fe43b4470ec8b240916b1637ae71f1944"} Oct 29 23:06:58.410090 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:58.408838 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd" event={"ID":"e5272969-d30b-4e39-93b3-587d000ef056","Type":"ContainerStarted","Data":"a1765b3af47f9da3758ddf47f64a8208f37ae048bc871a5fc60fa377a0c830c3"} Oct 29 23:06:58.410090 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:58.409328 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd" Oct 29 23:06:58.412227 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:58.411922 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl" event={"ID":"6b843c0e-5b58-46ef-b5cd-712b4acad831","Type":"ContainerStarted","Data":"c65ff59275a4316f041040164c9b9e8a05ff36241be2f6acbcaac8aaea7a3ed1"} Oct 29 23:06:58.412227 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:58.411979 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl" event={"ID":"6b843c0e-5b58-46ef-b5cd-712b4acad831","Type":"ContainerStarted","Data":"33d8c947f1ed647b98c276a642d1bc5219a403fda45129d756af19a2026926bb"} Oct 29 23:06:58.412227 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:58.412215 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl" Oct 29 23:06:58.484214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:58.483628 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd" podStartSLOduration=2.483607778 podStartE2EDuration="2.483607778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:58.481599402 +0000 UTC m=+1465.133097933" watchObservedRunningTime="2026-10-29 23:06:58.483607778 +0000 UTC m=+1465.135106315" Oct 29 23:06:58.552213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:58.543990 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl" podStartSLOduration=2.543965734 podStartE2EDuration="2.543965734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:58.527609508 +0000 UTC m=+1465.179108050" watchObservedRunningTime="2026-10-29 23:06:58.543965734 +0000 UTC m=+1465.195464265" Oct 29 23:06:58.690352 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:58.690317 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 29 23:06:58.690667 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:58.690635 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" podUID="bec61ff1-da16-48fd-97cc-7a6cc85213e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 29 23:06:59.607214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:59.604833 5057 generic.go:356] "Generic (PLEG): container finished" podID="84648cac-9c75-47a4-a3d6-df79e46b5fd9" containerID="5ddb2149716a2f78db3992a65ab20f55e0e78c90ed3af04eefa7b7d6f6c9c388" exitCode=0 Oct 29 23:06:59.615781 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:59.615745 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" event={"ID":"84648cac-9c75-47a4-a3d6-df79e46b5fd9","Type":"ContainerDied","Data":"5ddb2149716a2f78db3992a65ab20f55e0e78c90ed3af04eefa7b7d6f6c9c388"} Oct 29 23:06:59.744267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:59.696294 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 29 23:06:59.744267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:06:59.696346 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" podUID="bec61ff1-da16-48fd-97cc-7a6cc85213e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 29 23:07:00.754974 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.753616 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:07:00.762004 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.761712 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:07:00.782467 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.766636 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:07:00.804327 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.804215 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" event={"ID":"84648cac-9c75-47a4-a3d6-df79e46b5fd9","Type":"ContainerDied","Data":"796c8079892b6970579e9cb36f07a3b3dab7b5a2fb19ee1670d12d65183acd58"} Oct 29 23:07:00.804327 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.804285 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="796c8079892b6970579e9cb36f07a3b3dab7b5a2fb19ee1670d12d65183acd58" Oct 29 23:07:00.817645 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.817533 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 23:07:00.921420 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.921381 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-token\" (UniqueName: \"kubernetes.io/projected/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-token\") pod \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " Oct 29 23:07:00.921879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.921856 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-data\") pod \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " Oct 29 23:07:00.922151 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.922130 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istiod-ca-cert\") pod \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " Oct 29 23:07:00.922390 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.922374 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84648cac-9c75-47a4-a3d6-df79e46b5fd9-kube-api-access-mmthq\" (UniqueName: \"kubernetes.io/projected/84648cac-9c75-47a4-a3d6-df79e46b5fd9-kube-api-access-mmthq\") pod \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " Oct 29 23:07:00.922516 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.922503 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-credential-socket\") pod \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " Oct 29 23:07:00.931726 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.931692 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-workload-socket\") pod \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " Oct 29 23:07:00.931959 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.931943 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-workload-certs\") pod \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " Oct 29 23:07:00.932167 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.932150 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-envoy\") pod \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " Oct 29 23:07:00.932432 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.932413 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-podinfo\") pod \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\" (UID: \"84648cac-9c75-47a4-a3d6-df79e46b5fd9\") " Oct 29 23:07:00.935774 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.935747 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istiod-ca-cert" pod "84648cac-9c75-47a4-a3d6-df79e46b5fd9" (UID: "84648cac-9c75-47a4-a3d6-df79e46b5fd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:00.936144 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.936126 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-data" pod "84648cac-9c75-47a4-a3d6-df79e46b5fd9" (UID: "84648cac-9c75-47a4-a3d6-df79e46b5fd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:00.938794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.938761 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-workload-certs" pod "84648cac-9c75-47a4-a3d6-df79e46b5fd9" (UID: "84648cac-9c75-47a4-a3d6-df79e46b5fd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:00.939109 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.939091 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-workload-socket" pod "84648cac-9c75-47a4-a3d6-df79e46b5fd9" (UID: "84648cac-9c75-47a4-a3d6-df79e46b5fd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:00.939534 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.939516 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-credential-socket" pod "84648cac-9c75-47a4-a3d6-df79e46b5fd9" (UID: "84648cac-9c75-47a4-a3d6-df79e46b5fd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:00.997570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:00.997535 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-podinfo" pod "84648cac-9c75-47a4-a3d6-df79e46b5fd9" (UID: "84648cac-9c75-47a4-a3d6-df79e46b5fd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:01.097319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:01.049435 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-envoy" pod "84648cac-9c75-47a4-a3d6-df79e46b5fd9" (UID: "84648cac-9c75-47a4-a3d6-df79e46b5fd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.097319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:01.063731 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-token" pod "84648cac-9c75-47a4-a3d6-df79e46b5fd9" (UID: "84648cac-9c75-47a4-a3d6-df79e46b5fd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:01.097319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:01.086376 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84648cac-9c75-47a4-a3d6-df79e46b5fd9-kube-api-access-mmthq" pod "84648cac-9c75-47a4-a3d6-df79e46b5fd9" (UID: "84648cac-9c75-47a4-a3d6-df79e46b5fd9"). InnerVolumeSpecName "kube-api-access-mmthq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:01.097319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:01.035873 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.097319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:01.095369 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.097319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:01.095387 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.097319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:01.095405 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.097319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:01.095425 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.197514 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:01.197477 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.197514 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:01.197512 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.197514 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:01.197521 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmthq\" (UniqueName: \"kubernetes.io/projected/84648cac-9c75-47a4-a3d6-df79e46b5fd9-kube-api-access-mmthq\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.197781 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:01.197530 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84648cac-9c75-47a4-a3d6-df79e46b5fd9-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.812002 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:01.811967 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9" Oct 29 23:07:01.914086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:01.914047 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9"] Oct 29 23:07:01.973198 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:01.973142 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xgbc9"] Oct 29 23:07:03.486020 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:03.485985 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84648cac-9c75-47a4-a3d6-df79e46b5fd9" path="/var/lib/kubelet/pods/84648cac-9c75-47a4-a3d6-df79e46b5fd9/volumes" Oct 29 23:07:03.915969 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:07:03.915879 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:07:07.070955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:07.070919 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:07:09.067844 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:09.067805 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl"] Oct 29 23:07:09.068708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:09.068672 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" podUID="0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" containerName="istio-proxy" containerID="cri-o://187c6bdb9dc820e5d86851bd496c91dccb4b8594c6a46c68ff070ea5db833a15" gracePeriod=30 Oct 29 23:07:09.077673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:09.077210 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2"] Oct 29 23:07:09.077673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:09.077452 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" podUID="2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" containerName="istio-proxy" containerID="cri-o://2627aec871c546110e0c470c9df283c45a4b93ac99daca3be28c1ec29948fc0b" gracePeriod=30 Oct 29 23:07:09.608086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:09.608056 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd" Oct 29 23:07:09.608281 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:09.608142 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl" Oct 29 23:07:11.076859 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:11.076817 5057 generic.go:356] "Generic (PLEG): container finished" podID="34c31c14-402a-4cff-a9e1-0a6940ffc05a" containerID="b781f2aa745b0adda57e926bfbcc188a00e31148dfd3d50dba58ab4db389f26b" exitCode=0 Oct 29 23:07:11.077450 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:11.076890 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" event={"ID":"34c31c14-402a-4cff-a9e1-0a6940ffc05a","Type":"ContainerDied","Data":"b781f2aa745b0adda57e926bfbcc188a00e31148dfd3d50dba58ab4db389f26b"} Oct 29 23:07:11.077556 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:11.077538 5057 scope.go:122] "RemoveContainer" containerID="b781f2aa745b0adda57e926bfbcc188a00e31148dfd3d50dba58ab4db389f26b" Oct 29 23:07:11.079411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:11.079377 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:07:11.079553 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:11.079448 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:07:11.079553 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:11.079479 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:07:11.720290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:11.720253 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:07:12.086300 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:12.086219 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" event={"ID":"34c31c14-402a-4cff-a9e1-0a6940ffc05a","Type":"ContainerStarted","Data":"c5709440bc986a3e3743775769bfe6351e5d65476746ec9daeea3122b730a932"} Oct 29 23:07:12.097392 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:12.097368 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:07:12.098389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:12.098361 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 29 23:07:12.098510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:12.098401 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" podUID="34c31c14-402a-4cff-a9e1-0a6940ffc05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 29 23:07:13.098097 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:13.098060 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 29 23:07:13.098593 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:13.098128 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" podUID="34c31c14-402a-4cff-a9e1-0a6940ffc05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 29 23:07:14.098310 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.098276 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 29 23:07:14.098819 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.098334 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" podUID="34c31c14-402a-4cff-a9e1-0a6940ffc05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 29 23:07:14.692772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.692731 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:07:14.697839 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.697815 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:07:14.780954 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.780913 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istiod-ca-cert\") pod \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " Oct 29 23:07:14.781126 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.781055 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-workload-socket\") pod \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " Oct 29 23:07:14.781126 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.781111 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-podinfo\") pod \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " Oct 29 23:07:14.781362 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.781340 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-data\") pod \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " Oct 29 23:07:14.781458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.781415 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-workload-certs\") pod \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " Oct 29 23:07:14.781521 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.781473 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-token\" (UniqueName: \"kubernetes.io/projected/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-token\") pod \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " Oct 29 23:07:14.781580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.781526 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-credential-socket\") pod \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " Oct 29 23:07:14.781637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.781588 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-kube-api-access-pxhcf\" (UniqueName: \"kubernetes.io/projected/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-kube-api-access-pxhcf\") pod \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " Oct 29 23:07:14.781730 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.781712 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-envoy\") pod \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\" (UID: \"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992\") " Oct 29 23:07:14.783074 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.782992 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-credential-socket" pod "2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" (UID: "2bc4e0e4-9cb6-4804-aa8e-61ac50f73992"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:14.783269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.783244 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-data" pod "2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" (UID: "2bc4e0e4-9cb6-4804-aa8e-61ac50f73992"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:14.783482 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.783450 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istiod-ca-cert" pod "2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" (UID: "2bc4e0e4-9cb6-4804-aa8e-61ac50f73992"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:14.783699 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.783677 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-workload-socket" pod "2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" (UID: "2bc4e0e4-9cb6-4804-aa8e-61ac50f73992"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:14.783904 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.783885 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-workload-certs" pod "2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" (UID: "2bc4e0e4-9cb6-4804-aa8e-61ac50f73992"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:14.788976 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.788948 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-token" pod "2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" (UID: "2bc4e0e4-9cb6-4804-aa8e-61ac50f73992"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:14.789093 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.789057 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-envoy" pod "2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" (UID: "2bc4e0e4-9cb6-4804-aa8e-61ac50f73992"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:14.789525 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.789500 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-kube-api-access-pxhcf" pod "2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" (UID: "2bc4e0e4-9cb6-4804-aa8e-61ac50f73992"). InnerVolumeSpecName "kube-api-access-pxhcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:14.789622 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.789570 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-podinfo" pod "2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" (UID: "2bc4e0e4-9cb6-4804-aa8e-61ac50f73992"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:14.883068 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.883032 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-envoy\") pod \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " Oct 29 23:07:14.883321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.883112 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istiod-ca-cert\") pod \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " Oct 29 23:07:14.883321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.883172 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-token\" (UniqueName: \"kubernetes.io/projected/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-token\") pod \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " Oct 29 23:07:14.883914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.883876 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istiod-ca-cert" pod "0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" (UID: "0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:14.884325 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.884303 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-workload-certs\") pod \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " Oct 29 23:07:14.884451 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.884380 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-data\") pod \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " Oct 29 23:07:14.884451 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.884426 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-podinfo\") pod \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " Oct 29 23:07:14.884580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.884500 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-workload-socket\") pod \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " Oct 29 23:07:14.884727 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.884690 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-kube-api-access-xvjnf\" (UniqueName: \"kubernetes.io/projected/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-kube-api-access-xvjnf\") pod \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " Oct 29 23:07:14.885289 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.884786 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-credential-socket\") pod \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\" (UID: \"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb\") " Oct 29 23:07:14.885651 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.885628 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.885719 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.885657 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.885719 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.885671 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.885719 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.885683 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.885719 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.885696 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.885719 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.885709 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.885893 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.885723 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.885893 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.885735 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.885893 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.885748 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.885893 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.885761 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxhcf\" (UniqueName: \"kubernetes.io/projected/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992-kube-api-access-pxhcf\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.886022 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.885946 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-credential-socket" pod "0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" (UID: "0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:14.886126 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.886107 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-workload-certs" pod "0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" (UID: "0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:14.886297 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.886280 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-data" pod "0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" (UID: "0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:14.889528 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.889499 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-workload-socket" pod "0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" (UID: "0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:14.896361 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.896335 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-envoy" pod "0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" (UID: "0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:14.896490 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.896392 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-token" pod "0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" (UID: "0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:14.897421 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.897392 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-podinfo" pod "0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" (UID: "0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:14.898149 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.898123 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-kube-api-access-xvjnf" pod "0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" (UID: "0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb"). InnerVolumeSpecName "kube-api-access-xvjnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:14.987590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.987548 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:15.009349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.987598 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:15.009349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.987613 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:15.009349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.987627 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvjnf\" (UniqueName: \"kubernetes.io/projected/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-kube-api-access-xvjnf\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:15.009349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.987641 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:15.009349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.987666 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:15.009349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.987679 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:15.009349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:14.987691 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:15.107212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:15.103391 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 29 23:07:15.107212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:15.103530 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" podUID="34c31c14-402a-4cff-a9e1-0a6940ffc05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 29 23:07:15.267265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:15.247883 5057 generic.go:356] "Generic (PLEG): container finished" podID="2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" containerID="2627aec871c546110e0c470c9df283c45a4b93ac99daca3be28c1ec29948fc0b" exitCode=0 Oct 29 23:07:15.267265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:15.248253 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" event={"ID":"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992","Type":"ContainerDied","Data":"2627aec871c546110e0c470c9df283c45a4b93ac99daca3be28c1ec29948fc0b"} Oct 29 23:07:15.267265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:15.248376 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" event={"ID":"2bc4e0e4-9cb6-4804-aa8e-61ac50f73992","Type":"ContainerDied","Data":"888bb25ab3c5c9bbd08d6d6a6c94af0c3c83519e8700cda85aff20679b9b85a2"} Oct 29 23:07:15.267265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:15.248411 5057 scope.go:122] "RemoveContainer" containerID="2627aec871c546110e0c470c9df283c45a4b93ac99daca3be28c1ec29948fc0b" Oct 29 23:07:15.267265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:15.265317 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2" Oct 29 23:07:15.322400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:15.322359 5057 generic.go:356] "Generic (PLEG): container finished" podID="0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" containerID="187c6bdb9dc820e5d86851bd496c91dccb4b8594c6a46c68ff070ea5db833a15" exitCode=0 Oct 29 23:07:15.322723 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:15.322693 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" event={"ID":"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb","Type":"ContainerDied","Data":"187c6bdb9dc820e5d86851bd496c91dccb4b8594c6a46c68ff070ea5db833a15"} Oct 29 23:07:15.322901 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:15.322885 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" event={"ID":"0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb","Type":"ContainerDied","Data":"c88e073f155e25e39694077a1532d23709991b69b73ad009a0cebf935b7ed7cf"} Oct 29 23:07:15.323092 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:15.323079 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl" Oct 29 23:07:15.406553 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:15.405714 5057 scope.go:122] "RemoveContainer" containerID="2627aec871c546110e0c470c9df283c45a4b93ac99daca3be28c1ec29948fc0b" Oct 29 23:07:15.406553 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:07:15.406344 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2627aec871c546110e0c470c9df283c45a4b93ac99daca3be28c1ec29948fc0b\": container with ID starting with 2627aec871c546110e0c470c9df283c45a4b93ac99daca3be28c1ec29948fc0b not found: ID does not exist" containerID="2627aec871c546110e0c470c9df283c45a4b93ac99daca3be28c1ec29948fc0b" Oct 29 23:07:15.406553 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:15.406384 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2627aec871c546110e0c470c9df283c45a4b93ac99daca3be28c1ec29948fc0b"} err="failed to get container status \"2627aec871c546110e0c470c9df283c45a4b93ac99daca3be28c1ec29948fc0b\": rpc error: code = NotFound desc = could not find container \"2627aec871c546110e0c470c9df283c45a4b93ac99daca3be28c1ec29948fc0b\": container with ID starting with 2627aec871c546110e0c470c9df283c45a4b93ac99daca3be28c1ec29948fc0b not found: ID does not exist" Oct 29 23:07:15.406553 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:15.406408 5057 scope.go:122] "RemoveContainer" containerID="187c6bdb9dc820e5d86851bd496c91dccb4b8594c6a46c68ff070ea5db833a15" Oct 29 23:07:15.487960 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:15.487872 5057 scope.go:122] "RemoveContainer" containerID="187c6bdb9dc820e5d86851bd496c91dccb4b8594c6a46c68ff070ea5db833a15" Oct 29 23:07:15.489904 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:07:15.489874 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"187c6bdb9dc820e5d86851bd496c91dccb4b8594c6a46c68ff070ea5db833a15\": container with ID starting with 187c6bdb9dc820e5d86851bd496c91dccb4b8594c6a46c68ff070ea5db833a15 not found: ID does not exist" containerID="187c6bdb9dc820e5d86851bd496c91dccb4b8594c6a46c68ff070ea5db833a15" Oct 29 23:07:15.490150 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:15.490126 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"187c6bdb9dc820e5d86851bd496c91dccb4b8594c6a46c68ff070ea5db833a15"} err="failed to get container status \"187c6bdb9dc820e5d86851bd496c91dccb4b8594c6a46c68ff070ea5db833a15\": rpc error: code = NotFound desc = could not find container \"187c6bdb9dc820e5d86851bd496c91dccb4b8594c6a46c68ff070ea5db833a15\": container with ID starting with 187c6bdb9dc820e5d86851bd496c91dccb4b8594c6a46c68ff070ea5db833a15 not found: ID does not exist" Oct 29 23:07:15.492935 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:15.492914 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2"] Oct 29 23:07:15.493031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:15.492948 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-92xl2"] Oct 29 23:07:15.505897 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:15.505872 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl"] Oct 29 23:07:15.510019 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:15.509997 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-wf8cl"] Oct 29 23:07:16.099266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:16.098561 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 29 23:07:16.099266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:16.098625 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" podUID="34c31c14-402a-4cff-a9e1-0a6940ffc05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 29 23:07:17.097773 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:17.097732 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 29 23:07:17.098158 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:17.097791 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" podUID="34c31c14-402a-4cff-a9e1-0a6940ffc05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 29 23:07:17.486071 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:17.486029 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" path="/var/lib/kubelet/pods/0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb/volumes" Oct 29 23:07:17.486761 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:17.486737 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" path="/var/lib/kubelet/pods/2bc4e0e4-9cb6-4804-aa8e-61ac50f73992/volumes" Oct 29 23:07:17.790616 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:17.790530 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68"] Oct 29 23:07:17.790785 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:17.790725 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" containerID="cri-o://0e28d0bbf8ba9ed33195dd5ec647561118eec18c4fc31b859e1c483390b5b555" gracePeriod=30 Oct 29 23:07:18.097919 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:18.097831 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 29 23:07:18.099391 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:18.098577 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" podUID="34c31c14-402a-4cff-a9e1-0a6940ffc05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 29 23:07:19.102985 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:19.102953 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:07:19.103493 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:19.103383 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:07:19.104025 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:19.104005 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:07:20.787736 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:20.787701 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:07:20.788127 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:20.787770 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:07:23.095938 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.095912 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 23:07:23.228515 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.228479 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-token\" (UniqueName: \"kubernetes.io/projected/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-token\") pod \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " Oct 29 23:07:23.228738 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.228564 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-credential-socket\") pod \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " Oct 29 23:07:23.228738 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.228667 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istiod-ca-cert\") pod \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " Oct 29 23:07:23.228738 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.228701 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-podinfo\") pod \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " Oct 29 23:07:23.228924 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.228801 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-kube-api-access-gjrhc\" (UniqueName: \"kubernetes.io/projected/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-kube-api-access-gjrhc\") pod \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " Oct 29 23:07:23.228924 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.228903 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-workload-certs\") pod \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " Oct 29 23:07:23.229039 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.228953 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-workload-socket\") pod \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " Oct 29 23:07:23.229094 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.229064 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-envoy\") pod \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " Oct 29 23:07:23.229147 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.229112 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-data\") pod \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\" (UID: \"51d78fc8-72e3-468e-a7d5-b3ecdeaece75\") " Oct 29 23:07:23.229218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.229161 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istiod-ca-cert" pod "51d78fc8-72e3-468e-a7d5-b3ecdeaece75" (UID: "51d78fc8-72e3-468e-a7d5-b3ecdeaece75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:23.229453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.229429 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-credential-socket" pod "51d78fc8-72e3-468e-a7d5-b3ecdeaece75" (UID: "51d78fc8-72e3-468e-a7d5-b3ecdeaece75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:23.229898 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.229874 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-data" pod "51d78fc8-72e3-468e-a7d5-b3ecdeaece75" (UID: "51d78fc8-72e3-468e-a7d5-b3ecdeaece75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:23.230385 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.230151 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:23.230385 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.230177 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:23.230880 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.230606 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-workload-certs" pod "51d78fc8-72e3-468e-a7d5-b3ecdeaece75" (UID: "51d78fc8-72e3-468e-a7d5-b3ecdeaece75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:23.231551 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.231500 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-workload-socket" pod "51d78fc8-72e3-468e-a7d5-b3ecdeaece75" (UID: "51d78fc8-72e3-468e-a7d5-b3ecdeaece75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:23.233228 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.233147 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-podinfo" pod "51d78fc8-72e3-468e-a7d5-b3ecdeaece75" (UID: "51d78fc8-72e3-468e-a7d5-b3ecdeaece75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:23.234559 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.234523 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-envoy" pod "51d78fc8-72e3-468e-a7d5-b3ecdeaece75" (UID: "51d78fc8-72e3-468e-a7d5-b3ecdeaece75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:23.234665 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.234606 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-kube-api-access-gjrhc" pod "51d78fc8-72e3-468e-a7d5-b3ecdeaece75" (UID: "51d78fc8-72e3-468e-a7d5-b3ecdeaece75"). InnerVolumeSpecName "kube-api-access-gjrhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:23.247608 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.247538 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-token" pod "51d78fc8-72e3-468e-a7d5-b3ecdeaece75" (UID: "51d78fc8-72e3-468e-a7d5-b3ecdeaece75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:23.332220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.332085 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:23.332220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.332127 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:23.332220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.332141 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:23.332220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.332153 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:23.332220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.332166 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjrhc\" (UniqueName: \"kubernetes.io/projected/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-kube-api-access-gjrhc\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:23.332761 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.332180 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:23.332761 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.332265 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d78fc8-72e3-468e-a7d5-b3ecdeaece75-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:23.421493 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.421453 5057 generic.go:356] "Generic (PLEG): container finished" podID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerID="0e28d0bbf8ba9ed33195dd5ec647561118eec18c4fc31b859e1c483390b5b555" exitCode=0 Oct 29 23:07:23.421690 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.421511 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" Oct 29 23:07:23.421690 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.421543 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" event={"ID":"51d78fc8-72e3-468e-a7d5-b3ecdeaece75","Type":"ContainerDied","Data":"0e28d0bbf8ba9ed33195dd5ec647561118eec18c4fc31b859e1c483390b5b555"} Oct 29 23:07:23.421690 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.421638 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68" event={"ID":"51d78fc8-72e3-468e-a7d5-b3ecdeaece75","Type":"ContainerDied","Data":"0b7cf17088305928db707c5344e1d4b5948934d683d8ee80ea49caac042725d9"} Oct 29 23:07:23.421690 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.421669 5057 scope.go:122] "RemoveContainer" containerID="0e28d0bbf8ba9ed33195dd5ec647561118eec18c4fc31b859e1c483390b5b555" Oct 29 23:07:23.434744 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.434725 5057 scope.go:122] "RemoveContainer" containerID="0e28d0bbf8ba9ed33195dd5ec647561118eec18c4fc31b859e1c483390b5b555" Oct 29 23:07:23.434975 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:07:23.434958 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e28d0bbf8ba9ed33195dd5ec647561118eec18c4fc31b859e1c483390b5b555\": container with ID starting with 0e28d0bbf8ba9ed33195dd5ec647561118eec18c4fc31b859e1c483390b5b555 not found: ID does not exist" containerID="0e28d0bbf8ba9ed33195dd5ec647561118eec18c4fc31b859e1c483390b5b555" Oct 29 23:07:23.435041 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.434981 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e28d0bbf8ba9ed33195dd5ec647561118eec18c4fc31b859e1c483390b5b555"} err="failed to get container status \"0e28d0bbf8ba9ed33195dd5ec647561118eec18c4fc31b859e1c483390b5b555\": rpc error: code = NotFound desc = could not find container \"0e28d0bbf8ba9ed33195dd5ec647561118eec18c4fc31b859e1c483390b5b555\": container with ID starting with 0e28d0bbf8ba9ed33195dd5ec647561118eec18c4fc31b859e1c483390b5b555 not found: ID does not exist" Oct 29 23:07:23.487119 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.487092 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68"] Oct 29 23:07:23.495590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:23.495565 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-g9l68"] Oct 29 23:07:25.484073 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:25.484041 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" path="/var/lib/kubelet/pods/51d78fc8-72e3-468e-a7d5-b3ecdeaece75/volumes" Oct 29 23:07:29.264986 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:29.264952 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m"] Oct 29 23:07:29.265506 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:29.265231 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" podUID="90cef309-7998-4ca4-adff-acd2e5d5ffc2" containerName="maas-api" containerID="cri-o://f4208c5e34c71fc10bfc65d0aa810f8878a80973d437be915e0a492bd02be79c" gracePeriod=30 Oct 29 23:07:29.431262 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:29.431222 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh"] Oct 29 23:07:29.431478 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:29.431453 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh" podUID="55bfce1f-8524-43aa-949d-e933d41ca3cb" containerName="payload-processing" containerID="cri-o://a8a724a9f3104ff32e2cceecb47fd6fc7ac629032bb03173cc2e5fb2e2822652" gracePeriod=30 Oct 29 23:07:29.449599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:29.449565 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz"] Oct 29 23:07:29.450017 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:29.449964 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz" podUID="e3361dd3-0ed7-4b16-aeb4-17633004627a" containerName="payload-pre-processing" containerID="cri-o://368d495962b7d090507df19ab247f779bdc3f485d8525f7277aa715f4c4cba37" gracePeriod=30 Oct 29 23:07:29.549058 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:29.549024 5057 generic.go:356] "Generic (PLEG): container finished" podID="90cef309-7998-4ca4-adff-acd2e5d5ffc2" containerID="f4208c5e34c71fc10bfc65d0aa810f8878a80973d437be915e0a492bd02be79c" exitCode=0 Oct 29 23:07:29.550212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:29.549376 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" event={"ID":"90cef309-7998-4ca4-adff-acd2e5d5ffc2","Type":"ContainerDied","Data":"f4208c5e34c71fc10bfc65d0aa810f8878a80973d437be915e0a492bd02be79c"} Oct 29 23:07:29.712364 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:29.712331 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" Oct 29 23:07:29.722784 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:29.722750 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz" podUID="e3361dd3-0ed7-4b16-aeb4-17633004627a" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.167:9004: connect: connection refused" Oct 29 23:07:29.846198 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:29.846045 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90cef309-7998-4ca4-adff-acd2e5d5ffc2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cef309-7998-4ca4-adff-acd2e5d5ffc2-metrics-certs\") pod \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\" (UID: \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\") " Oct 29 23:07:29.848911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:29.848321 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90cef309-7998-4ca4-adff-acd2e5d5ffc2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90cef309-7998-4ca4-adff-acd2e5d5ffc2-openshift-service-ca\") pod \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\" (UID: \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\") " Oct 29 23:07:29.848911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:29.848429 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90cef309-7998-4ca4-adff-acd2e5d5ffc2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90cef309-7998-4ca4-adff-acd2e5d5ffc2-maas-api-tls\") pod \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\" (UID: \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\") " Oct 29 23:07:29.848911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:29.848609 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90cef309-7998-4ca4-adff-acd2e5d5ffc2-kube-api-access-sk7hh\" (UniqueName: \"kubernetes.io/projected/90cef309-7998-4ca4-adff-acd2e5d5ffc2-kube-api-access-sk7hh\") pod \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\" (UID: \"90cef309-7998-4ca4-adff-acd2e5d5ffc2\") " Oct 29 23:07:29.850884 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:29.850853 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90cef309-7998-4ca4-adff-acd2e5d5ffc2-openshift-service-ca" pod "90cef309-7998-4ca4-adff-acd2e5d5ffc2" (UID: "90cef309-7998-4ca4-adff-acd2e5d5ffc2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:29.859293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:29.858951 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90cef309-7998-4ca4-adff-acd2e5d5ffc2-maas-api-tls" pod "90cef309-7998-4ca4-adff-acd2e5d5ffc2" (UID: "90cef309-7998-4ca4-adff-acd2e5d5ffc2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:29.859293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:29.858947 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90cef309-7998-4ca4-adff-acd2e5d5ffc2-kube-api-access-sk7hh" pod "90cef309-7998-4ca4-adff-acd2e5d5ffc2" (UID: "90cef309-7998-4ca4-adff-acd2e5d5ffc2"). InnerVolumeSpecName "kube-api-access-sk7hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:29.862233 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:29.862165 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90cef309-7998-4ca4-adff-acd2e5d5ffc2-metrics-certs" pod "90cef309-7998-4ca4-adff-acd2e5d5ffc2" (UID: "90cef309-7998-4ca4-adff-acd2e5d5ffc2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:29.873401 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:29.873371 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh" Oct 29 23:07:30.029234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.027478 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55bfce1f-8524-43aa-949d-e933d41ca3cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bfce1f-8524-43aa-949d-e933d41ca3cb-plugins-config-volume\") pod \"55bfce1f-8524-43aa-949d-e933d41ca3cb\" (UID: \"55bfce1f-8524-43aa-949d-e933d41ca3cb\") " Oct 29 23:07:30.029234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.027594 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55bfce1f-8524-43aa-949d-e933d41ca3cb-kube-api-access-z9d94\" (UniqueName: \"kubernetes.io/projected/55bfce1f-8524-43aa-949d-e933d41ca3cb-kube-api-access-z9d94\") pod \"55bfce1f-8524-43aa-949d-e933d41ca3cb\" (UID: \"55bfce1f-8524-43aa-949d-e933d41ca3cb\") " Oct 29 23:07:30.112344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.076762 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cef309-7998-4ca4-adff-acd2e5d5ffc2-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:30.112344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.076796 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90cef309-7998-4ca4-adff-acd2e5d5ffc2-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:30.112344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.076811 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90cef309-7998-4ca4-adff-acd2e5d5ffc2-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:30.112344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.076826 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk7hh\" (UniqueName: \"kubernetes.io/projected/90cef309-7998-4ca4-adff-acd2e5d5ffc2-kube-api-access-sk7hh\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:30.166560 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.166516 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55bfce1f-8524-43aa-949d-e933d41ca3cb-kube-api-access-z9d94" pod "55bfce1f-8524-43aa-949d-e933d41ca3cb" (UID: "55bfce1f-8524-43aa-949d-e933d41ca3cb"). InnerVolumeSpecName "kube-api-access-z9d94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:30.240426 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.240378 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9d94\" (UniqueName: \"kubernetes.io/projected/55bfce1f-8524-43aa-949d-e933d41ca3cb-kube-api-access-z9d94\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:30.333242 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.318851 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55bfce1f-8524-43aa-949d-e933d41ca3cb-plugins-config-volume" pod "55bfce1f-8524-43aa-949d-e933d41ca3cb" (UID: "55bfce1f-8524-43aa-949d-e933d41ca3cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:30.341611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.341574 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bfce1f-8524-43aa-949d-e933d41ca3cb-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:30.354135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.354110 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz" Oct 29 23:07:30.546356 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.546324 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3361dd3-0ed7-4b16-aeb4-17633004627a-kube-api-access-vkjrs\" (UniqueName: \"kubernetes.io/projected/e3361dd3-0ed7-4b16-aeb4-17633004627a-kube-api-access-vkjrs\") pod \"e3361dd3-0ed7-4b16-aeb4-17633004627a\" (UID: \"e3361dd3-0ed7-4b16-aeb4-17633004627a\") " Oct 29 23:07:30.546564 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.546540 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3361dd3-0ed7-4b16-aeb4-17633004627a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3361dd3-0ed7-4b16-aeb4-17633004627a-plugins-config-volume\") pod \"e3361dd3-0ed7-4b16-aeb4-17633004627a\" (UID: \"e3361dd3-0ed7-4b16-aeb4-17633004627a\") " Oct 29 23:07:30.546918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.546896 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3361dd3-0ed7-4b16-aeb4-17633004627a-plugins-config-volume" pod "e3361dd3-0ed7-4b16-aeb4-17633004627a" (UID: "e3361dd3-0ed7-4b16-aeb4-17633004627a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:30.547228 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.547212 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3361dd3-0ed7-4b16-aeb4-17633004627a-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:30.548447 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.548416 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3361dd3-0ed7-4b16-aeb4-17633004627a-kube-api-access-vkjrs" pod "e3361dd3-0ed7-4b16-aeb4-17633004627a" (UID: "e3361dd3-0ed7-4b16-aeb4-17633004627a"). InnerVolumeSpecName "kube-api-access-vkjrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:30.561344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.561321 5057 generic.go:356] "Generic (PLEG): container finished" podID="55bfce1f-8524-43aa-949d-e933d41ca3cb" containerID="a8a724a9f3104ff32e2cceecb47fd6fc7ac629032bb03173cc2e5fb2e2822652" exitCode=0 Oct 29 23:07:30.561495 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.561387 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh" Oct 29 23:07:30.561495 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.561423 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh" event={"ID":"55bfce1f-8524-43aa-949d-e933d41ca3cb","Type":"ContainerDied","Data":"a8a724a9f3104ff32e2cceecb47fd6fc7ac629032bb03173cc2e5fb2e2822652"} Oct 29 23:07:30.561495 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.561490 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh" event={"ID":"55bfce1f-8524-43aa-949d-e933d41ca3cb","Type":"ContainerDied","Data":"fb9d3c758594003d896edfac494c281aaed0cb01a5d3fe0e1810f3d63f919a55"} Oct 29 23:07:30.561634 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.561528 5057 scope.go:122] "RemoveContainer" containerID="a8a724a9f3104ff32e2cceecb47fd6fc7ac629032bb03173cc2e5fb2e2822652" Oct 29 23:07:30.564969 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.564802 5057 generic.go:356] "Generic (PLEG): container finished" podID="e3361dd3-0ed7-4b16-aeb4-17633004627a" containerID="368d495962b7d090507df19ab247f779bdc3f485d8525f7277aa715f4c4cba37" exitCode=0 Oct 29 23:07:30.564969 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.564922 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz" event={"ID":"e3361dd3-0ed7-4b16-aeb4-17633004627a","Type":"ContainerDied","Data":"368d495962b7d090507df19ab247f779bdc3f485d8525f7277aa715f4c4cba37"} Oct 29 23:07:30.564969 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.564967 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz" event={"ID":"e3361dd3-0ed7-4b16-aeb4-17633004627a","Type":"ContainerDied","Data":"ff6032ed36dce654b2f5cbb95d63c42b5fbe754b311522bd2e9119a1727ace95"} Oct 29 23:07:30.565169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.565036 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz" Oct 29 23:07:30.571607 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.571569 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" event={"ID":"90cef309-7998-4ca4-adff-acd2e5d5ffc2","Type":"ContainerDied","Data":"199026090e2480476f74d0cfecaadbaa4b47c28f095e125402919baf01015bce"} Oct 29 23:07:30.571721 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.571667 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m" Oct 29 23:07:30.579067 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.579046 5057 scope.go:122] "RemoveContainer" containerID="a8a724a9f3104ff32e2cceecb47fd6fc7ac629032bb03173cc2e5fb2e2822652" Oct 29 23:07:30.580454 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:07:30.579484 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a724a9f3104ff32e2cceecb47fd6fc7ac629032bb03173cc2e5fb2e2822652\": container with ID starting with a8a724a9f3104ff32e2cceecb47fd6fc7ac629032bb03173cc2e5fb2e2822652 not found: ID does not exist" containerID="a8a724a9f3104ff32e2cceecb47fd6fc7ac629032bb03173cc2e5fb2e2822652" Oct 29 23:07:30.580454 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.579526 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a724a9f3104ff32e2cceecb47fd6fc7ac629032bb03173cc2e5fb2e2822652"} err="failed to get container status \"a8a724a9f3104ff32e2cceecb47fd6fc7ac629032bb03173cc2e5fb2e2822652\": rpc error: code = NotFound desc = could not find container \"a8a724a9f3104ff32e2cceecb47fd6fc7ac629032bb03173cc2e5fb2e2822652\": container with ID starting with a8a724a9f3104ff32e2cceecb47fd6fc7ac629032bb03173cc2e5fb2e2822652 not found: ID does not exist" Oct 29 23:07:30.580454 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.579550 5057 scope.go:122] "RemoveContainer" containerID="368d495962b7d090507df19ab247f779bdc3f485d8525f7277aa715f4c4cba37" Oct 29 23:07:30.592806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.592782 5057 scope.go:122] "RemoveContainer" containerID="368d495962b7d090507df19ab247f779bdc3f485d8525f7277aa715f4c4cba37" Oct 29 23:07:30.593220 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:07:30.593158 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368d495962b7d090507df19ab247f779bdc3f485d8525f7277aa715f4c4cba37\": container with ID starting with 368d495962b7d090507df19ab247f779bdc3f485d8525f7277aa715f4c4cba37 not found: ID does not exist" containerID="368d495962b7d090507df19ab247f779bdc3f485d8525f7277aa715f4c4cba37" Oct 29 23:07:30.593324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.593296 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368d495962b7d090507df19ab247f779bdc3f485d8525f7277aa715f4c4cba37"} err="failed to get container status \"368d495962b7d090507df19ab247f779bdc3f485d8525f7277aa715f4c4cba37\": rpc error: code = NotFound desc = could not find container \"368d495962b7d090507df19ab247f779bdc3f485d8525f7277aa715f4c4cba37\": container with ID starting with 368d495962b7d090507df19ab247f779bdc3f485d8525f7277aa715f4c4cba37 not found: ID does not exist" Oct 29 23:07:30.593399 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.593325 5057 scope.go:122] "RemoveContainer" containerID="f4208c5e34c71fc10bfc65d0aa810f8878a80973d437be915e0a492bd02be79c" Oct 29 23:07:30.642202 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.642154 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh"] Oct 29 23:07:30.648985 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.648951 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkjrs\" (UniqueName: \"kubernetes.io/projected/e3361dd3-0ed7-4b16-aeb4-17633004627a-kube-api-access-vkjrs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:30.649130 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.649039 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh"] Oct 29 23:07:30.700484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.700455 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz"] Oct 29 23:07:30.704547 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.704521 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-6f4dfd9bf9-qwppz"] Oct 29 23:07:30.722997 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.722963 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-64ccf55db-fq5wh" podUID="55bfce1f-8524-43aa-949d-e933d41ca3cb" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.166:9004: i/o timeout" Oct 29 23:07:30.744058 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.744026 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m"] Oct 29 23:07:30.750906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:30.750876 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-41b12c-7d46d44d9b-pzk4m"] Oct 29 23:07:31.492669 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:31.492632 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55bfce1f-8524-43aa-949d-e933d41ca3cb" path="/var/lib/kubelet/pods/55bfce1f-8524-43aa-949d-e933d41ca3cb/volumes" Oct 29 23:07:31.493166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:31.493150 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90cef309-7998-4ca4-adff-acd2e5d5ffc2" path="/var/lib/kubelet/pods/90cef309-7998-4ca4-adff-acd2e5d5ffc2/volumes" Oct 29 23:07:31.493634 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:31.493619 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3361dd3-0ed7-4b16-aeb4-17633004627a" path="/var/lib/kubelet/pods/e3361dd3-0ed7-4b16-aeb4-17633004627a/volumes" Oct 29 23:07:34.097209 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:07:34.090261 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:07:34.757028 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:34.756998 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:07:34.797627 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:34.797596 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:07:36.048737 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:36.048707 5057 scope.go:122] "RemoveContainer" containerID="5ddb2149716a2f78db3992a65ab20f55e0e78c90ed3af04eefa7b7d6f6c9c388" Oct 29 23:07:38.015765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:38.015658 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9"] Oct 29 23:07:38.016611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:38.016574 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" podUID="0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" containerName="istio-proxy" containerID="cri-o://fe0c5862222696fdebba6329b776e5b27a29da4eb00973d7336bdfd4716f8d08" gracePeriod=30 Oct 29 23:07:38.029078 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:38.029040 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl"] Oct 29 23:07:38.029415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:38.029362 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" podUID="7d0a6630-350e-4f6d-9c7f-c261cc545ea5" containerName="maas-api" containerID="cri-o://c5298ae8cb8b7fc1caf9bd2578f0796b8386381dca8d8ac29b7ee1bb4cc17f2e" gracePeriod=30 Oct 29 23:07:38.890211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:38.883279 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:07:38.890211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:38.883712 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" podUID="0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:07:38.964548 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:38.964462 5057 generic.go:356] "Generic (PLEG): container finished" podID="7d0a6630-350e-4f6d-9c7f-c261cc545ea5" containerID="c5298ae8cb8b7fc1caf9bd2578f0796b8386381dca8d8ac29b7ee1bb4cc17f2e" exitCode=0 Oct 29 23:07:38.964718 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:38.964672 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" event={"ID":"7d0a6630-350e-4f6d-9c7f-c261cc545ea5","Type":"ContainerDied","Data":"c5298ae8cb8b7fc1caf9bd2578f0796b8386381dca8d8ac29b7ee1bb4cc17f2e"} Oct 29 23:07:38.964793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:38.964775 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" event={"ID":"7d0a6630-350e-4f6d-9c7f-c261cc545ea5","Type":"ContainerDied","Data":"d49bb51727c03833559b638593974f320ed99122c2bd755b8c693d387c424982"} Oct 29 23:07:38.964852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:38.964812 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d49bb51727c03833559b638593974f320ed99122c2bd755b8c693d387c424982" Oct 29 23:07:38.976221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:38.967371 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" Oct 29 23:07:39.088649 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:39.088613 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-openshift-service-ca\") pod \"7d0a6630-350e-4f6d-9c7f-c261cc545ea5\" (UID: \"7d0a6630-350e-4f6d-9c7f-c261cc545ea5\") " Oct 29 23:07:39.089131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:39.088944 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-maas-api-tls\") pod \"7d0a6630-350e-4f6d-9c7f-c261cc545ea5\" (UID: \"7d0a6630-350e-4f6d-9c7f-c261cc545ea5\") " Oct 29 23:07:39.089131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:39.089074 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-kube-api-access-8n2sk\" (UniqueName: \"kubernetes.io/projected/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-kube-api-access-8n2sk\") pod \"7d0a6630-350e-4f6d-9c7f-c261cc545ea5\" (UID: \"7d0a6630-350e-4f6d-9c7f-c261cc545ea5\") " Oct 29 23:07:39.089269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:39.089215 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-metrics-certs\") pod \"7d0a6630-350e-4f6d-9c7f-c261cc545ea5\" (UID: \"7d0a6630-350e-4f6d-9c7f-c261cc545ea5\") " Oct 29 23:07:39.096748 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:39.096718 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-openshift-service-ca" pod "7d0a6630-350e-4f6d-9c7f-c261cc545ea5" (UID: "7d0a6630-350e-4f6d-9c7f-c261cc545ea5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:39.097371 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:39.097250 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-metrics-certs" pod "7d0a6630-350e-4f6d-9c7f-c261cc545ea5" (UID: "7d0a6630-350e-4f6d-9c7f-c261cc545ea5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:39.097371 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:39.097335 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-maas-api-tls" pod "7d0a6630-350e-4f6d-9c7f-c261cc545ea5" (UID: "7d0a6630-350e-4f6d-9c7f-c261cc545ea5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:39.097550 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:39.097417 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-kube-api-access-8n2sk" pod "7d0a6630-350e-4f6d-9c7f-c261cc545ea5" (UID: "7d0a6630-350e-4f6d-9c7f-c261cc545ea5"). InnerVolumeSpecName "kube-api-access-8n2sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:39.192257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:39.192218 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:39.192257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:39.192260 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:39.192632 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:39.192275 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:39.192632 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:39.192287 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n2sk\" (UniqueName: \"kubernetes.io/projected/7d0a6630-350e-4f6d-9c7f-c261cc545ea5-kube-api-access-8n2sk\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:39.972320 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:39.972286 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl" Oct 29 23:07:40.065207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.059583 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl"] Oct 29 23:07:40.066525 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.066363 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0c72c-5f88d4bd-lsqxl"] Oct 29 23:07:40.111606 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.111574 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm"] Oct 29 23:07:40.112527 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112502 5057 cpu_manager.go:414] "containerMap: removing container" podUID="55bfce1f-8524-43aa-949d-e933d41ca3cb" containerName="payload-processing" Oct 29 23:07:40.112527 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112526 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55bfce1f-8524-43aa-949d-e933d41ca3cb" containerName="payload-processing" Oct 29 23:07:40.112707 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112542 5057 cpu_manager.go:414] "containerMap: removing container" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" Oct 29 23:07:40.112707 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112549 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" Oct 29 23:07:40.112707 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112563 5057 cpu_manager.go:414] "containerMap: removing container" podUID="90cef309-7998-4ca4-adff-acd2e5d5ffc2" containerName="maas-api" Oct 29 23:07:40.112707 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112571 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90cef309-7998-4ca4-adff-acd2e5d5ffc2" containerName="maas-api" Oct 29 23:07:40.112707 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112606 5057 cpu_manager.go:414] "containerMap: removing container" podUID="2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" containerName="istio-proxy" Oct 29 23:07:40.112707 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112614 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" containerName="istio-proxy" Oct 29 23:07:40.112707 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112629 5057 cpu_manager.go:414] "containerMap: removing container" podUID="0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" containerName="istio-proxy" Oct 29 23:07:40.112707 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112636 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" containerName="istio-proxy" Oct 29 23:07:40.112707 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112655 5057 cpu_manager.go:414] "containerMap: removing container" podUID="84648cac-9c75-47a4-a3d6-df79e46b5fd9" containerName="istio-proxy" Oct 29 23:07:40.112707 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112663 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84648cac-9c75-47a4-a3d6-df79e46b5fd9" containerName="istio-proxy" Oct 29 23:07:40.112707 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112686 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7d0a6630-350e-4f6d-9c7f-c261cc545ea5" containerName="maas-api" Oct 29 23:07:40.112707 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112693 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d0a6630-350e-4f6d-9c7f-c261cc545ea5" containerName="maas-api" Oct 29 23:07:40.113285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112713 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e3361dd3-0ed7-4b16-aeb4-17633004627a" containerName="payload-pre-processing" Oct 29 23:07:40.113285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112720 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3361dd3-0ed7-4b16-aeb4-17633004627a" containerName="payload-pre-processing" Oct 29 23:07:40.113285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112904 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bc4e0e4-9cb6-4804-aa8e-61ac50f73992" containerName="istio-proxy" Oct 29 23:07:40.113285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112916 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7fa998-f3a7-4ef0-85f2-8662c3fe8bbb" containerName="istio-proxy" Oct 29 23:07:40.113285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112926 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="84648cac-9c75-47a4-a3d6-df79e46b5fd9" containerName="istio-proxy" Oct 29 23:07:40.113285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112938 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d0a6630-350e-4f6d-9c7f-c261cc545ea5" containerName="maas-api" Oct 29 23:07:40.113285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112949 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3361dd3-0ed7-4b16-aeb4-17633004627a" containerName="payload-pre-processing" Oct 29 23:07:40.113285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112966 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="55bfce1f-8524-43aa-949d-e933d41ca3cb" containerName="payload-processing" Oct 29 23:07:40.113285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112975 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d78fc8-72e3-468e-a7d5-b3ecdeaece75" containerName="istio-proxy" Oct 29 23:07:40.113285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.112986 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="90cef309-7998-4ca4-adff-acd2e5d5ffc2" containerName="maas-api" Oct 29 23:07:40.116063 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.116043 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.225266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.190355 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-6d21fd-openshift-default-dockercfg-xpxhl\"" Oct 29 23:07:40.360268 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.198628 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm"] Oct 29 23:07:40.360594 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.226676 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-podinfo\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.397409 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.397371 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-data\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.397689 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.397667 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-envoy\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.401321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.401292 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-token\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.401520 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.401498 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istiod-ca-cert\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.405737 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.405711 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmfgk\" (UniqueName: \"kubernetes.io/projected/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-kube-api-access-cmfgk\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.405964 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.405946 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-workload-certs\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.406198 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.406168 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-workload-socket\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.434398 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.434358 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-credential-socket\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.604265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.575360 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-workload-certs\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.604663 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.604627 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-workload-socket\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.611400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.611372 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-credential-socket\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.611555 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.611537 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-podinfo\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.611700 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.611684 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-data\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.611781 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.611755 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-envoy\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.611849 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.611816 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-token\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.611906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.611847 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istiod-ca-cert\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.611906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.611889 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmfgk\" (UniqueName: \"kubernetes.io/projected/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-kube-api-access-cmfgk\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.612294 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.605164 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-workload-certs\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.612596 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.612576 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-credential-socket\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.612687 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.606592 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-workload-socket\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.627249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.613569 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-data\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.628670 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.628639 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istiod-ca-cert\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.629536 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.629451 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-podinfo\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.678215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.657839 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-envoy\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.680111 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.680079 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-token\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.681335 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.681304 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmfgk\" (UniqueName: \"kubernetes.io/projected/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-kube-api-access-cmfgk\") pod \"e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:40.950995 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:40.950962 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:41.288811 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:07:41.288781 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f96e8f2_d75a_4a6d_a3a8_08f7f0007a33.slice/crio-86b195beae22aabac90259825fc9a21b9e909f30fafa51398a2764388729f0d0 WatchSource:0}: Error finding container 86b195beae22aabac90259825fc9a21b9e909f30fafa51398a2764388729f0d0: Status 404 returned error can't find the container with id 86b195beae22aabac90259825fc9a21b9e909f30fafa51398a2764388729f0d0 Oct 29 23:07:41.289160 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:41.289135 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm"] Oct 29 23:07:41.291533 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:41.291496 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:07:41.291618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:41.291562 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:07:41.291618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:41.291591 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:07:41.483730 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:41.483689 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d0a6630-350e-4f6d-9c7f-c261cc545ea5" path="/var/lib/kubelet/pods/7d0a6630-350e-4f6d-9c7f-c261cc545ea5/volumes" Oct 29 23:07:41.762210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:41.761217 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq"] Oct 29 23:07:41.762210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:41.761485 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" podUID="7d0539b1-a376-4016-900e-382ad30b3e02" containerName="istio-proxy" containerID="cri-o://d06f6ced558bd847df8ff89d304b0582ec70ae7bcacd5f6afc77241cfdc5562e" gracePeriod=30 Oct 29 23:07:41.991485 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:41.991447 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" event={"ID":"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33","Type":"ContainerStarted","Data":"027e0c714b6b67a7b970559717a1af03d6908446225b0b4bf54126b5cf7df370"} Oct 29 23:07:41.991636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:41.991511 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" event={"ID":"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33","Type":"ContainerStarted","Data":"86b195beae22aabac90259825fc9a21b9e909f30fafa51398a2764388729f0d0"} Oct 29 23:07:42.045305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:42.045099 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" podStartSLOduration=2.045085858 podStartE2EDuration="2.045085858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:42.039728551 +0000 UTC m=+1508.691227081" watchObservedRunningTime="2026-10-29 23:07:42.045085858 +0000 UTC m=+1508.696584387" Oct 29 23:07:42.951592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:42.951547 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:42.958460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:42.958434 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:42.999559 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:42.999531 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:43.000530 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.000509 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:07:43.832570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.832512 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:07:43.929179 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.929135 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-credential-socket\") pod \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " Oct 29 23:07:43.929408 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.929332 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-envoy\") pod \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " Oct 29 23:07:43.929480 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.929423 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-workload-socket\") pod \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " Oct 29 23:07:43.929480 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.929442 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-credential-socket" pod "0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" (UID: "0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:43.929480 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.929463 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istiod-ca-cert\") pod \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " Oct 29 23:07:43.929649 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.929541 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-podinfo\") pod \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " Oct 29 23:07:43.929649 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.929638 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-token\" (UniqueName: \"kubernetes.io/projected/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-token\") pod \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " Oct 29 23:07:43.929763 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.929668 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-workload-socket" pod "0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" (UID: "0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:43.929763 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.929736 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-kube-api-access-ztpvs\" (UniqueName: \"kubernetes.io/projected/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-kube-api-access-ztpvs\") pod \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " Oct 29 23:07:43.929873 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.929794 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-data\") pod \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " Oct 29 23:07:43.929939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.929874 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-workload-certs\") pod \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\" (UID: \"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1\") " Oct 29 23:07:43.929992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.929942 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istiod-ca-cert" pod "0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" (UID: "0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:43.930331 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.930299 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-data" pod "0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" (UID: "0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:43.930538 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.930515 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-workload-certs" pod "0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" (UID: "0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:43.930755 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.930732 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:43.930829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.930759 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:43.930829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.930772 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:43.930829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.930785 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:43.933084 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.933061 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-podinfo" pod "0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" (UID: "0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:43.933176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.933134 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-kube-api-access-ztpvs" pod "0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" (UID: "0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1"). InnerVolumeSpecName "kube-api-access-ztpvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:43.933618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.933587 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-envoy" pod "0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" (UID: "0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:43.933618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:43.933588 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-token" pod "0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" (UID: "0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:44.008931 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:44.008898 5057 generic.go:356] "Generic (PLEG): container finished" podID="0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" containerID="fe0c5862222696fdebba6329b776e5b27a29da4eb00973d7336bdfd4716f8d08" exitCode=0 Oct 29 23:07:44.009387 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:44.008947 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" Oct 29 23:07:44.009387 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:44.008980 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" event={"ID":"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1","Type":"ContainerDied","Data":"fe0c5862222696fdebba6329b776e5b27a29da4eb00973d7336bdfd4716f8d08"} Oct 29 23:07:44.009387 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:44.009033 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9" event={"ID":"0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1","Type":"ContainerDied","Data":"12ec086bbdf7e391a124000d5ca38f40982ea4325f150f0fc27d9e9c71e7c756"} Oct 29 23:07:44.009387 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:44.009056 5057 scope.go:122] "RemoveContainer" containerID="fe0c5862222696fdebba6329b776e5b27a29da4eb00973d7336bdfd4716f8d08" Oct 29 23:07:44.021282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:44.021263 5057 scope.go:122] "RemoveContainer" containerID="e4e8a3003c3c34e7083d941ab9221ee7feefdcae06a8aefa92c99f5038b1a4ff" Oct 29 23:07:44.032087 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:44.032062 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:44.032311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:44.032090 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:44.032311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:44.032104 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:44.032311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:44.032116 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztpvs\" (UniqueName: \"kubernetes.io/projected/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-kube-api-access-ztpvs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:44.032311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:44.032131 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:44.034126 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:44.034103 5057 scope.go:122] "RemoveContainer" containerID="fe0c5862222696fdebba6329b776e5b27a29da4eb00973d7336bdfd4716f8d08" Oct 29 23:07:44.034458 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:07:44.034437 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe0c5862222696fdebba6329b776e5b27a29da4eb00973d7336bdfd4716f8d08\": container with ID starting with fe0c5862222696fdebba6329b776e5b27a29da4eb00973d7336bdfd4716f8d08 not found: ID does not exist" containerID="fe0c5862222696fdebba6329b776e5b27a29da4eb00973d7336bdfd4716f8d08" Oct 29 23:07:44.034560 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:44.034464 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe0c5862222696fdebba6329b776e5b27a29da4eb00973d7336bdfd4716f8d08"} err="failed to get container status \"fe0c5862222696fdebba6329b776e5b27a29da4eb00973d7336bdfd4716f8d08\": rpc error: code = NotFound desc = could not find container \"fe0c5862222696fdebba6329b776e5b27a29da4eb00973d7336bdfd4716f8d08\": container with ID starting with fe0c5862222696fdebba6329b776e5b27a29da4eb00973d7336bdfd4716f8d08 not found: ID does not exist" Oct 29 23:07:44.034560 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:44.034483 5057 scope.go:122] "RemoveContainer" containerID="e4e8a3003c3c34e7083d941ab9221ee7feefdcae06a8aefa92c99f5038b1a4ff" Oct 29 23:07:44.034772 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:07:44.034740 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e8a3003c3c34e7083d941ab9221ee7feefdcae06a8aefa92c99f5038b1a4ff\": container with ID starting with e4e8a3003c3c34e7083d941ab9221ee7feefdcae06a8aefa92c99f5038b1a4ff not found: ID does not exist" containerID="e4e8a3003c3c34e7083d941ab9221ee7feefdcae06a8aefa92c99f5038b1a4ff" Oct 29 23:07:44.034861 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:44.034773 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e8a3003c3c34e7083d941ab9221ee7feefdcae06a8aefa92c99f5038b1a4ff"} err="failed to get container status \"e4e8a3003c3c34e7083d941ab9221ee7feefdcae06a8aefa92c99f5038b1a4ff\": rpc error: code = NotFound desc = could not find container \"e4e8a3003c3c34e7083d941ab9221ee7feefdcae06a8aefa92c99f5038b1a4ff\": container with ID starting with e4e8a3003c3c34e7083d941ab9221ee7feefdcae06a8aefa92c99f5038b1a4ff not found: ID does not exist" Oct 29 23:07:44.073922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:44.073863 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9"] Oct 29 23:07:44.078495 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:44.078468 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-41b12c-openshift-default-7d454488c5-cbrs9"] Oct 29 23:07:45.484726 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:45.484691 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" path="/var/lib/kubelet/pods/0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1/volumes" Oct 29 23:07:46.259700 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.259661 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc"] Oct 29 23:07:46.260483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.260465 5057 cpu_manager.go:414] "containerMap: removing container" podUID="0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" containerName="istio-proxy" Oct 29 23:07:46.260483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.260481 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" containerName="istio-proxy" Oct 29 23:07:46.260629 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.260494 5057 cpu_manager.go:414] "containerMap: removing container" podUID="0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" containerName="istio-proxy" Oct 29 23:07:46.260629 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.260500 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" containerName="istio-proxy" Oct 29 23:07:46.260629 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.260621 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" containerName="istio-proxy" Oct 29 23:07:46.260629 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.260628 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9c9cf6-5558-4e14-a1c1-1ca61f40e3c1" containerName="istio-proxy" Oct 29 23:07:46.264599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.264577 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc" Oct 29 23:07:46.281996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.281974 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272e697e-08f1-44a5-838b-742a0f570253-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc\" (UID: \"272e697e-08f1-44a5-838b-742a0f570253\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc" Oct 29 23:07:46.282087 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.282029 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5xf6\" (UniqueName: \"kubernetes.io/projected/272e697e-08f1-44a5-838b-742a0f570253-kube-api-access-n5xf6\") pod \"payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc\" (UID: \"272e697e-08f1-44a5-838b-742a0f570253\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc" Oct 29 23:07:46.289604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.289581 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc"] Oct 29 23:07:46.305231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.299135 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-6d21fd\"" Oct 29 23:07:46.305947 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.305922 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t"] Oct 29 23:07:46.309986 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.309967 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t" Oct 29 23:07:46.357217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.352548 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-6d21fd-dockercfg-m5s2f\"" Oct 29 23:07:46.378235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.373859 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t"] Oct 29 23:07:46.687220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.687163 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5xf6\" (UniqueName: \"kubernetes.io/projected/272e697e-08f1-44a5-838b-742a0f570253-kube-api-access-n5xf6\") pod \"payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc\" (UID: \"272e697e-08f1-44a5-838b-742a0f570253\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc" Oct 29 23:07:46.689500 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.689469 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb12844-8806-48b3-af64-ec85f9f77eec-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t\" (UID: \"9bb12844-8806-48b3-af64-ec85f9f77eec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t" Oct 29 23:07:46.723271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.723233 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrmd2\" (UniqueName: \"kubernetes.io/projected/9bb12844-8806-48b3-af64-ec85f9f77eec-kube-api-access-mrmd2\") pod \"payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t\" (UID: \"9bb12844-8806-48b3-af64-ec85f9f77eec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t" Oct 29 23:07:46.724454 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.724419 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272e697e-08f1-44a5-838b-742a0f570253-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc\" (UID: \"272e697e-08f1-44a5-838b-742a0f570253\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc" Oct 29 23:07:46.731417 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.731372 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272e697e-08f1-44a5-838b-742a0f570253-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc\" (UID: \"272e697e-08f1-44a5-838b-742a0f570253\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc" Oct 29 23:07:46.809822 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.803234 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5xf6\" (UniqueName: \"kubernetes.io/projected/272e697e-08f1-44a5-838b-742a0f570253-kube-api-access-n5xf6\") pod \"payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc\" (UID: \"272e697e-08f1-44a5-838b-742a0f570253\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc" Oct 29 23:07:46.829202 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.829151 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrmd2\" (UniqueName: \"kubernetes.io/projected/9bb12844-8806-48b3-af64-ec85f9f77eec-kube-api-access-mrmd2\") pod \"payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t\" (UID: \"9bb12844-8806-48b3-af64-ec85f9f77eec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t" Oct 29 23:07:46.829413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.829393 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb12844-8806-48b3-af64-ec85f9f77eec-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t\" (UID: \"9bb12844-8806-48b3-af64-ec85f9f77eec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t" Oct 29 23:07:46.830379 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.830347 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb12844-8806-48b3-af64-ec85f9f77eec-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t\" (UID: \"9bb12844-8806-48b3-af64-ec85f9f77eec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t" Oct 29 23:07:46.855996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.855960 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrmd2\" (UniqueName: \"kubernetes.io/projected/9bb12844-8806-48b3-af64-ec85f9f77eec-kube-api-access-mrmd2\") pod \"payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t\" (UID: \"9bb12844-8806-48b3-af64-ec85f9f77eec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t" Oct 29 23:07:46.878055 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.878026 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc" Oct 29 23:07:46.929235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:46.929028 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t" Oct 29 23:07:47.071913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.071771 5057 generic.go:356] "Generic (PLEG): container finished" podID="7d0539b1-a376-4016-900e-382ad30b3e02" containerID="d06f6ced558bd847df8ff89d304b0582ec70ae7bcacd5f6afc77241cfdc5562e" exitCode=0 Oct 29 23:07:47.071913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.071862 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" event={"ID":"7d0539b1-a376-4016-900e-382ad30b3e02","Type":"ContainerDied","Data":"d06f6ced558bd847df8ff89d304b0582ec70ae7bcacd5f6afc77241cfdc5562e"} Oct 29 23:07:47.204488 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.204267 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 23:07:47.238524 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.237647 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-istio-envoy\") pod \"7d0539b1-a376-4016-900e-382ad30b3e02\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " Oct 29 23:07:47.238524 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.238082 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-credential-socket\") pod \"7d0539b1-a376-4016-900e-382ad30b3e02\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " Oct 29 23:07:47.238524 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.238153 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0539b1-a376-4016-900e-382ad30b3e02-istio-token\" (UniqueName: \"kubernetes.io/projected/7d0539b1-a376-4016-900e-382ad30b3e02-istio-token\") pod \"7d0539b1-a376-4016-900e-382ad30b3e02\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " Oct 29 23:07:47.238524 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.238308 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d0539b1-a376-4016-900e-382ad30b3e02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d0539b1-a376-4016-900e-382ad30b3e02-istiod-ca-cert\") pod \"7d0539b1-a376-4016-900e-382ad30b3e02\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " Oct 29 23:07:47.238524 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.238406 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d0539b1-a376-4016-900e-382ad30b3e02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d0539b1-a376-4016-900e-382ad30b3e02-istio-podinfo\") pod \"7d0539b1-a376-4016-900e-382ad30b3e02\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " Oct 29 23:07:47.238524 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.238518 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-istio-data\") pod \"7d0539b1-a376-4016-900e-382ad30b3e02\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " Oct 29 23:07:47.238972 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.238541 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-workload-socket\") pod \"7d0539b1-a376-4016-900e-382ad30b3e02\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " Oct 29 23:07:47.238972 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.238722 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-workload-certs\") pod \"7d0539b1-a376-4016-900e-382ad30b3e02\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " Oct 29 23:07:47.238972 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.238842 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0539b1-a376-4016-900e-382ad30b3e02-kube-api-access-c9j2t\" (UniqueName: \"kubernetes.io/projected/7d0539b1-a376-4016-900e-382ad30b3e02-kube-api-access-c9j2t\") pod \"7d0539b1-a376-4016-900e-382ad30b3e02\" (UID: \"7d0539b1-a376-4016-900e-382ad30b3e02\") " Oct 29 23:07:47.239946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.239417 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-istio-data" pod "7d0539b1-a376-4016-900e-382ad30b3e02" (UID: "7d0539b1-a376-4016-900e-382ad30b3e02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:47.239946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.239606 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-workload-socket" pod "7d0539b1-a376-4016-900e-382ad30b3e02" (UID: "7d0539b1-a376-4016-900e-382ad30b3e02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:47.239946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.239615 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:47.240178 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.240010 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0539b1-a376-4016-900e-382ad30b3e02-istiod-ca-cert" pod "7d0539b1-a376-4016-900e-382ad30b3e02" (UID: "7d0539b1-a376-4016-900e-382ad30b3e02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:47.240178 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.240073 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-credential-socket" pod "7d0539b1-a376-4016-900e-382ad30b3e02" (UID: "7d0539b1-a376-4016-900e-382ad30b3e02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:47.240178 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.240120 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-workload-certs" pod "7d0539b1-a376-4016-900e-382ad30b3e02" (UID: "7d0539b1-a376-4016-900e-382ad30b3e02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:47.243223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.242640 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0539b1-a376-4016-900e-382ad30b3e02-kube-api-access-c9j2t" pod "7d0539b1-a376-4016-900e-382ad30b3e02" (UID: "7d0539b1-a376-4016-900e-382ad30b3e02"). InnerVolumeSpecName "kube-api-access-c9j2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:47.244866 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.244835 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d0539b1-a376-4016-900e-382ad30b3e02-istio-podinfo" pod "7d0539b1-a376-4016-900e-382ad30b3e02" (UID: "7d0539b1-a376-4016-900e-382ad30b3e02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:47.245428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.245402 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-istio-envoy" pod "7d0539b1-a376-4016-900e-382ad30b3e02" (UID: "7d0539b1-a376-4016-900e-382ad30b3e02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:47.246716 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.246685 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0539b1-a376-4016-900e-382ad30b3e02-istio-token" pod "7d0539b1-a376-4016-900e-382ad30b3e02" (UID: "7d0539b1-a376-4016-900e-382ad30b3e02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:47.350510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.350478 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9j2t\" (UniqueName: \"kubernetes.io/projected/7d0539b1-a376-4016-900e-382ad30b3e02-kube-api-access-c9j2t\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:47.350510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.350514 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:47.350717 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.350530 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:47.350717 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.350544 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d0539b1-a376-4016-900e-382ad30b3e02-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:47.350717 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.350560 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d0539b1-a376-4016-900e-382ad30b3e02-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:47.350717 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.350576 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d0539b1-a376-4016-900e-382ad30b3e02-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:47.350717 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.350588 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:47.352343 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.352316 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d0539b1-a376-4016-900e-382ad30b3e02-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:47.378824 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.378789 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv"] Oct 29 23:07:47.379101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.379072 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" podUID="d728ed56-8f9a-4396-8386-7ce602570421" containerName="istio-proxy" containerID="cri-o://b8ace4e57b9d7bae23977abf99140415de27b909d27661a88eed3d25debf9dcc" gracePeriod=30 Oct 29 23:07:47.385687 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.384260 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk"] Oct 29 23:07:47.385687 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.384458 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" podUID="927fbba6-8dbc-4e71-81d9-de5b2859ba5d" containerName="istio-proxy" containerID="cri-o://b06c63c268e8707a35327eb1844c311f17243f2a54cb0457315c6bf72956b402" gracePeriod=30 Oct 29 23:07:47.415545 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:07:47.415515 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272e697e_08f1_44a5_838b_742a0f570253.slice/crio-4d3e3f4875de5b8f04b8a2034ad23966f80c1cc3ef2081e05e3f4fda6b0a67d6 WatchSource:0}: Error finding container 4d3e3f4875de5b8f04b8a2034ad23966f80c1cc3ef2081e05e3f4fda6b0a67d6: Status 404 returned error can't find the container with id 4d3e3f4875de5b8f04b8a2034ad23966f80c1cc3ef2081e05e3f4fda6b0a67d6 Oct 29 23:07:47.418348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.418323 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc"] Oct 29 23:07:47.680025 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:47.674484 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t"] Oct 29 23:07:48.085527 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:48.085430 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t" event={"ID":"9bb12844-8806-48b3-af64-ec85f9f77eec","Type":"ContainerStarted","Data":"c8d19fff5b0e03cc1d89b822139f5314ba74ef94e650aeeebfc43cddb9ac1af6"} Oct 29 23:07:48.085527 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:48.085499 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t" event={"ID":"9bb12844-8806-48b3-af64-ec85f9f77eec","Type":"ContainerStarted","Data":"7cf7b08a6d29cdd290919af7d7ab1481c98dd1fc010e95dd90d510ad7667ea95"} Oct 29 23:07:48.085527 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:48.085526 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t" Oct 29 23:07:48.087057 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:48.087034 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" Oct 29 23:07:48.087173 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:48.087058 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq" event={"ID":"7d0539b1-a376-4016-900e-382ad30b3e02","Type":"ContainerDied","Data":"109a03d5784597ae94f91f9b0dcf0058a97a4cdfbf0719650c3f872f7853f59b"} Oct 29 23:07:48.087173 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:48.087105 5057 scope.go:122] "RemoveContainer" containerID="d06f6ced558bd847df8ff89d304b0582ec70ae7bcacd5f6afc77241cfdc5562e" Oct 29 23:07:48.088900 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:48.088877 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc" event={"ID":"272e697e-08f1-44a5-838b-742a0f570253","Type":"ContainerStarted","Data":"a621481b6a8bd3c554b67fdb33f63b6c48b6c5db0d32a080eda91f5c27d7e63d"} Oct 29 23:07:48.088963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:48.088928 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc" event={"ID":"272e697e-08f1-44a5-838b-742a0f570253","Type":"ContainerStarted","Data":"4d3e3f4875de5b8f04b8a2034ad23966f80c1cc3ef2081e05e3f4fda6b0a67d6"} Oct 29 23:07:48.089122 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:48.089108 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc" Oct 29 23:07:48.132515 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:48.132453 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t" podStartSLOduration=2.132435535 podStartE2EDuration="2.132435535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:48.128773121 +0000 UTC m=+1514.780271652" watchObservedRunningTime="2026-10-29 23:07:48.132435535 +0000 UTC m=+1514.783934069" Oct 29 23:07:48.172678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:48.172640 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq"] Oct 29 23:07:48.177727 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:48.177696 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qlqgq"] Oct 29 23:07:48.228908 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:48.228840 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc" podStartSLOduration=2.228819563 podStartE2EDuration="2.228819563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:48.222503348 +0000 UTC m=+1514.874001878" watchObservedRunningTime="2026-10-29 23:07:48.228819563 +0000 UTC m=+1514.880318095" Oct 29 23:07:48.670368 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:48.670331 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:07:49.483674 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:49.483638 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d0539b1-a376-4016-900e-382ad30b3e02" path="/var/lib/kubelet/pods/7d0539b1-a376-4016-900e-382ad30b3e02/volumes" Oct 29 23:07:50.532976 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:50.532938 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc"] Oct 29 23:07:50.561140 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:50.560513 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8"] Oct 29 23:07:52.916224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:52.911929 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:07:53.017156 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.017120 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d728ed56-8f9a-4396-8386-7ce602570421-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d728ed56-8f9a-4396-8386-7ce602570421-istiod-ca-cert\") pod \"d728ed56-8f9a-4396-8386-7ce602570421\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " Oct 29 23:07:53.017388 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.017302 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-credential-socket\") pod \"d728ed56-8f9a-4396-8386-7ce602570421\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " Oct 29 23:07:53.017469 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.017418 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-workload-certs\") pod \"d728ed56-8f9a-4396-8386-7ce602570421\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " Oct 29 23:07:53.017537 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.017478 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-workload-socket\") pod \"d728ed56-8f9a-4396-8386-7ce602570421\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " Oct 29 23:07:53.017995 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.017961 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d728ed56-8f9a-4396-8386-7ce602570421-istio-token\" (UniqueName: \"kubernetes.io/projected/d728ed56-8f9a-4396-8386-7ce602570421-istio-token\") pod \"d728ed56-8f9a-4396-8386-7ce602570421\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " Oct 29 23:07:53.018105 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.018070 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d728ed56-8f9a-4396-8386-7ce602570421-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d728ed56-8f9a-4396-8386-7ce602570421-istio-podinfo\") pod \"d728ed56-8f9a-4396-8386-7ce602570421\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " Oct 29 23:07:53.018182 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.018168 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-istio-data\") pod \"d728ed56-8f9a-4396-8386-7ce602570421\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " Oct 29 23:07:53.018521 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.018259 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-istio-envoy\") pod \"d728ed56-8f9a-4396-8386-7ce602570421\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " Oct 29 23:07:53.018521 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.018297 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d728ed56-8f9a-4396-8386-7ce602570421-kube-api-access-7p6jg\" (UniqueName: \"kubernetes.io/projected/d728ed56-8f9a-4396-8386-7ce602570421-kube-api-access-7p6jg\") pod \"d728ed56-8f9a-4396-8386-7ce602570421\" (UID: \"d728ed56-8f9a-4396-8386-7ce602570421\") " Oct 29 23:07:53.019479 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.018765 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-workload-socket" pod "d728ed56-8f9a-4396-8386-7ce602570421" (UID: "d728ed56-8f9a-4396-8386-7ce602570421"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:53.019479 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.019223 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.020538 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.020353 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-credential-socket" pod "d728ed56-8f9a-4396-8386-7ce602570421" (UID: "d728ed56-8f9a-4396-8386-7ce602570421"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:53.022084 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.021707 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-istio-data" pod "d728ed56-8f9a-4396-8386-7ce602570421" (UID: "d728ed56-8f9a-4396-8386-7ce602570421"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:53.022623 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.022261 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-workload-certs" pod "d728ed56-8f9a-4396-8386-7ce602570421" (UID: "d728ed56-8f9a-4396-8386-7ce602570421"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:53.023636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.023592 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d728ed56-8f9a-4396-8386-7ce602570421-istiod-ca-cert" pod "d728ed56-8f9a-4396-8386-7ce602570421" (UID: "d728ed56-8f9a-4396-8386-7ce602570421"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:53.101435 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.097709 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d728ed56-8f9a-4396-8386-7ce602570421-istio-podinfo" pod "d728ed56-8f9a-4396-8386-7ce602570421" (UID: "d728ed56-8f9a-4396-8386-7ce602570421"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:53.104902 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.104853 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d728ed56-8f9a-4396-8386-7ce602570421-kube-api-access-7p6jg" pod "d728ed56-8f9a-4396-8386-7ce602570421" (UID: "d728ed56-8f9a-4396-8386-7ce602570421"). InnerVolumeSpecName "kube-api-access-7p6jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:53.117489 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.117295 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-istio-envoy" pod "d728ed56-8f9a-4396-8386-7ce602570421" (UID: "d728ed56-8f9a-4396-8386-7ce602570421"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:53.117489 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.117436 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d728ed56-8f9a-4396-8386-7ce602570421-istio-token" pod "d728ed56-8f9a-4396-8386-7ce602570421" (UID: "d728ed56-8f9a-4396-8386-7ce602570421"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:53.159081 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.158732 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:07:53.203945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.202106 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d728ed56-8f9a-4396-8386-7ce602570421-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.203945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.202132 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d728ed56-8f9a-4396-8386-7ce602570421-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.203945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.202140 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.203945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.202150 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.203945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.202162 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p6jg\" (UniqueName: \"kubernetes.io/projected/d728ed56-8f9a-4396-8386-7ce602570421-kube-api-access-7p6jg\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.203945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.202170 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d728ed56-8f9a-4396-8386-7ce602570421-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.203945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.202179 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.203945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.202229 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d728ed56-8f9a-4396-8386-7ce602570421-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.302762 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.302716 5057 generic.go:356] "Generic (PLEG): container finished" podID="d728ed56-8f9a-4396-8386-7ce602570421" containerID="b8ace4e57b9d7bae23977abf99140415de27b909d27661a88eed3d25debf9dcc" exitCode=0 Oct 29 23:07:53.302934 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.302800 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" event={"ID":"d728ed56-8f9a-4396-8386-7ce602570421","Type":"ContainerDied","Data":"b8ace4e57b9d7bae23977abf99140415de27b909d27661a88eed3d25debf9dcc"} Oct 29 23:07:53.302934 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.302854 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" event={"ID":"d728ed56-8f9a-4396-8386-7ce602570421","Type":"ContainerDied","Data":"704b6730b81486f41ae80b330e25e375252dd8c86925c2ca7ccc8360cb4940fe"} Oct 29 23:07:53.302934 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.302879 5057 scope.go:122] "RemoveContainer" containerID="b8ace4e57b9d7bae23977abf99140415de27b909d27661a88eed3d25debf9dcc" Oct 29 23:07:53.303144 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.303018 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv" Oct 29 23:07:53.307324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.306905 5057 generic.go:356] "Generic (PLEG): container finished" podID="927fbba6-8dbc-4e71-81d9-de5b2859ba5d" containerID="b06c63c268e8707a35327eb1844c311f17243f2a54cb0457315c6bf72956b402" exitCode=0 Oct 29 23:07:53.307324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.306954 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" event={"ID":"927fbba6-8dbc-4e71-81d9-de5b2859ba5d","Type":"ContainerDied","Data":"b06c63c268e8707a35327eb1844c311f17243f2a54cb0457315c6bf72956b402"} Oct 29 23:07:53.307324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.306995 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" event={"ID":"927fbba6-8dbc-4e71-81d9-de5b2859ba5d","Type":"ContainerDied","Data":"7c4e2ac423f85eb57f15e89b541692b6c2d7cc3dfd30ef8df77049ebe2b380ab"} Oct 29 23:07:53.307324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.307062 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk" Oct 29 23:07:53.309401 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.309294 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istiod-ca-cert\") pod \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " Oct 29 23:07:53.309487 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.309420 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-workload-socket\") pod \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " Oct 29 23:07:53.309550 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.309513 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-podinfo\") pod \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " Oct 29 23:07:53.309606 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.309573 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-kube-api-access-jcnx5\" (UniqueName: \"kubernetes.io/projected/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-kube-api-access-jcnx5\") pod \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " Oct 29 23:07:53.309672 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.309645 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-credential-socket\") pod \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " Oct 29 23:07:53.309757 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.309734 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-token\" (UniqueName: \"kubernetes.io/projected/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-token\") pod \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " Oct 29 23:07:53.309827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.309810 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-workload-certs\") pod \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " Oct 29 23:07:53.309919 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.309904 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-data\") pod \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " Oct 29 23:07:53.309987 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.309949 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-workload-socket" pod "927fbba6-8dbc-4e71-81d9-de5b2859ba5d" (UID: "927fbba6-8dbc-4e71-81d9-de5b2859ba5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:53.318570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.310100 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-envoy\") pod \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\" (UID: \"927fbba6-8dbc-4e71-81d9-de5b2859ba5d\") " Oct 29 23:07:53.318570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.310213 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-data" pod "927fbba6-8dbc-4e71-81d9-de5b2859ba5d" (UID: "927fbba6-8dbc-4e71-81d9-de5b2859ba5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:53.318570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.312708 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-podinfo" pod "927fbba6-8dbc-4e71-81d9-de5b2859ba5d" (UID: "927fbba6-8dbc-4e71-81d9-de5b2859ba5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:53.318570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.315136 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-kube-api-access-jcnx5" pod "927fbba6-8dbc-4e71-81d9-de5b2859ba5d" (UID: "927fbba6-8dbc-4e71-81d9-de5b2859ba5d"). InnerVolumeSpecName "kube-api-access-jcnx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:53.318570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.315364 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-workload-certs" pod "927fbba6-8dbc-4e71-81d9-de5b2859ba5d" (UID: "927fbba6-8dbc-4e71-81d9-de5b2859ba5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:53.318570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.315634 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istiod-ca-cert" pod "927fbba6-8dbc-4e71-81d9-de5b2859ba5d" (UID: "927fbba6-8dbc-4e71-81d9-de5b2859ba5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:53.318570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.315861 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-credential-socket" pod "927fbba6-8dbc-4e71-81d9-de5b2859ba5d" (UID: "927fbba6-8dbc-4e71-81d9-de5b2859ba5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:53.318570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.318402 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.318570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.318424 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.318570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.318438 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcnx5\" (UniqueName: \"kubernetes.io/projected/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-kube-api-access-jcnx5\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.318570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.318466 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.318570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.318479 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.318570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.318492 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.318570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.318505 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.319494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.319237 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-token" pod "927fbba6-8dbc-4e71-81d9-de5b2859ba5d" (UID: "927fbba6-8dbc-4e71-81d9-de5b2859ba5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:53.319494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.319356 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-envoy" pod "927fbba6-8dbc-4e71-81d9-de5b2859ba5d" (UID: "927fbba6-8dbc-4e71-81d9-de5b2859ba5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:53.398057 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.398029 5057 scope.go:122] "RemoveContainer" containerID="18425919557b62eb641feaa3a65c07a14d9d83af576c81ff516db8896d16151d" Oct 29 23:07:53.419896 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.419841 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.419896 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.419860 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927fbba6-8dbc-4e71-81d9-de5b2859ba5d-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.442831 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.420774 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv"] Oct 29 23:07:53.684210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.669508 5057 scope.go:122] "RemoveContainer" containerID="b8ace4e57b9d7bae23977abf99140415de27b909d27661a88eed3d25debf9dcc" Oct 29 23:07:53.713213 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:07:53.702788 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8ace4e57b9d7bae23977abf99140415de27b909d27661a88eed3d25debf9dcc\": container with ID starting with b8ace4e57b9d7bae23977abf99140415de27b909d27661a88eed3d25debf9dcc not found: ID does not exist" containerID="b8ace4e57b9d7bae23977abf99140415de27b909d27661a88eed3d25debf9dcc" Oct 29 23:07:53.713213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.702835 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ace4e57b9d7bae23977abf99140415de27b909d27661a88eed3d25debf9dcc"} err="failed to get container status \"b8ace4e57b9d7bae23977abf99140415de27b909d27661a88eed3d25debf9dcc\": rpc error: code = NotFound desc = could not find container \"b8ace4e57b9d7bae23977abf99140415de27b909d27661a88eed3d25debf9dcc\": container with ID starting with b8ace4e57b9d7bae23977abf99140415de27b909d27661a88eed3d25debf9dcc not found: ID does not exist" Oct 29 23:07:53.713213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.702861 5057 scope.go:122] "RemoveContainer" containerID="18425919557b62eb641feaa3a65c07a14d9d83af576c81ff516db8896d16151d" Oct 29 23:07:53.713213 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:07:53.705307 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18425919557b62eb641feaa3a65c07a14d9d83af576c81ff516db8896d16151d\": container with ID starting with 18425919557b62eb641feaa3a65c07a14d9d83af576c81ff516db8896d16151d not found: ID does not exist" containerID="18425919557b62eb641feaa3a65c07a14d9d83af576c81ff516db8896d16151d" Oct 29 23:07:53.713213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.705344 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18425919557b62eb641feaa3a65c07a14d9d83af576c81ff516db8896d16151d"} err="failed to get container status \"18425919557b62eb641feaa3a65c07a14d9d83af576c81ff516db8896d16151d\": rpc error: code = NotFound desc = could not find container \"18425919557b62eb641feaa3a65c07a14d9d83af576c81ff516db8896d16151d\": container with ID starting with 18425919557b62eb641feaa3a65c07a14d9d83af576c81ff516db8896d16151d not found: ID does not exist" Oct 29 23:07:53.713213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.705367 5057 scope.go:122] "RemoveContainer" containerID="b06c63c268e8707a35327eb1844c311f17243f2a54cb0457315c6bf72956b402" Oct 29 23:07:53.752211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.749127 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-v4jwv"] Oct 29 23:07:53.791154 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.789836 5057 scope.go:122] "RemoveContainer" containerID="b06c63c268e8707a35327eb1844c311f17243f2a54cb0457315c6bf72956b402" Oct 29 23:07:53.798145 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:07:53.797960 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b06c63c268e8707a35327eb1844c311f17243f2a54cb0457315c6bf72956b402\": container with ID starting with b06c63c268e8707a35327eb1844c311f17243f2a54cb0457315c6bf72956b402 not found: ID does not exist" containerID="b06c63c268e8707a35327eb1844c311f17243f2a54cb0457315c6bf72956b402" Oct 29 23:07:53.798290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.798150 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06c63c268e8707a35327eb1844c311f17243f2a54cb0457315c6bf72956b402"} err="failed to get container status \"b06c63c268e8707a35327eb1844c311f17243f2a54cb0457315c6bf72956b402\": rpc error: code = NotFound desc = could not find container \"b06c63c268e8707a35327eb1844c311f17243f2a54cb0457315c6bf72956b402\": container with ID starting with b06c63c268e8707a35327eb1844c311f17243f2a54cb0457315c6bf72956b402 not found: ID does not exist" Oct 29 23:07:53.890270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.889373 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk"] Oct 29 23:07:53.890270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:53.889436 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-2llsk"] Oct 29 23:07:55.499515 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:55.499477 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="927fbba6-8dbc-4e71-81d9-de5b2859ba5d" path="/var/lib/kubelet/pods/927fbba6-8dbc-4e71-81d9-de5b2859ba5d/volumes" Oct 29 23:07:55.500043 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:55.500028 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d728ed56-8f9a-4396-8386-7ce602570421" path="/var/lib/kubelet/pods/d728ed56-8f9a-4396-8386-7ce602570421/volumes" Oct 29 23:07:55.540211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:55.537244 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc" podUID="5fa0b407-b14b-47c2-a6d7-a16536017857" containerName="payload-processing" containerID="cri-o://3ef171e02c2b37132106337f7542134a572ad231a7c510ee817d0a40579e69a2" gracePeriod=25 Oct 29 23:07:55.573276 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:55.564121 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8" podUID="41347d77-4ec9-4e28-8651-6868480061c4" containerName="payload-pre-processing" containerID="cri-o://1d4c57ca2fb3f6789f3964c8b38d46e58225853e87355ff3575ddb7da50e8885" gracePeriod=25 Oct 29 23:07:56.403392 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.403362 5057 generic.go:356] "Generic (PLEG): container finished" podID="41347d77-4ec9-4e28-8651-6868480061c4" containerID="1d4c57ca2fb3f6789f3964c8b38d46e58225853e87355ff3575ddb7da50e8885" exitCode=0 Oct 29 23:07:56.403537 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.403448 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8" event={"ID":"41347d77-4ec9-4e28-8651-6868480061c4","Type":"ContainerDied","Data":"1d4c57ca2fb3f6789f3964c8b38d46e58225853e87355ff3575ddb7da50e8885"} Oct 29 23:07:56.404085 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.404067 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc" Oct 29 23:07:56.404847 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.404831 5057 generic.go:356] "Generic (PLEG): container finished" podID="5fa0b407-b14b-47c2-a6d7-a16536017857" containerID="3ef171e02c2b37132106337f7542134a572ad231a7c510ee817d0a40579e69a2" exitCode=0 Oct 29 23:07:56.404954 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.404873 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc" event={"ID":"5fa0b407-b14b-47c2-a6d7-a16536017857","Type":"ContainerDied","Data":"3ef171e02c2b37132106337f7542134a572ad231a7c510ee817d0a40579e69a2"} Oct 29 23:07:56.404954 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.404908 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc" event={"ID":"5fa0b407-b14b-47c2-a6d7-a16536017857","Type":"ContainerDied","Data":"54d9f49641fed6a1dd609d8fc9dd40f566949795c86d4d33fc73969ad7a73d60"} Oct 29 23:07:56.404954 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.404928 5057 scope.go:122] "RemoveContainer" containerID="3ef171e02c2b37132106337f7542134a572ad231a7c510ee817d0a40579e69a2" Oct 29 23:07:56.470260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.467031 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8" Oct 29 23:07:56.497429 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.497405 5057 scope.go:122] "RemoveContainer" containerID="3ef171e02c2b37132106337f7542134a572ad231a7c510ee817d0a40579e69a2" Oct 29 23:07:56.500687 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:07:56.500056 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ef171e02c2b37132106337f7542134a572ad231a7c510ee817d0a40579e69a2\": container with ID starting with 3ef171e02c2b37132106337f7542134a572ad231a7c510ee817d0a40579e69a2 not found: ID does not exist" containerID="3ef171e02c2b37132106337f7542134a572ad231a7c510ee817d0a40579e69a2" Oct 29 23:07:56.500687 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.500097 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ef171e02c2b37132106337f7542134a572ad231a7c510ee817d0a40579e69a2"} err="failed to get container status \"3ef171e02c2b37132106337f7542134a572ad231a7c510ee817d0a40579e69a2\": rpc error: code = NotFound desc = could not find container \"3ef171e02c2b37132106337f7542134a572ad231a7c510ee817d0a40579e69a2\": container with ID starting with 3ef171e02c2b37132106337f7542134a572ad231a7c510ee817d0a40579e69a2 not found: ID does not exist" Oct 29 23:07:56.503614 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.503268 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41347d77-4ec9-4e28-8651-6868480061c4-kube-api-access-9tnxd\" (UniqueName: \"kubernetes.io/projected/41347d77-4ec9-4e28-8651-6868480061c4-kube-api-access-9tnxd\") pod \"41347d77-4ec9-4e28-8651-6868480061c4\" (UID: \"41347d77-4ec9-4e28-8651-6868480061c4\") " Oct 29 23:07:56.503614 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.503361 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fa0b407-b14b-47c2-a6d7-a16536017857-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fa0b407-b14b-47c2-a6d7-a16536017857-plugins-config-volume\") pod \"5fa0b407-b14b-47c2-a6d7-a16536017857\" (UID: \"5fa0b407-b14b-47c2-a6d7-a16536017857\") " Oct 29 23:07:56.503614 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.503468 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41347d77-4ec9-4e28-8651-6868480061c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41347d77-4ec9-4e28-8651-6868480061c4-plugins-config-volume\") pod \"41347d77-4ec9-4e28-8651-6868480061c4\" (UID: \"41347d77-4ec9-4e28-8651-6868480061c4\") " Oct 29 23:07:56.503614 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.503528 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa0b407-b14b-47c2-a6d7-a16536017857-kube-api-access-m2v6f\" (UniqueName: \"kubernetes.io/projected/5fa0b407-b14b-47c2-a6d7-a16536017857-kube-api-access-m2v6f\") pod \"5fa0b407-b14b-47c2-a6d7-a16536017857\" (UID: \"5fa0b407-b14b-47c2-a6d7-a16536017857\") " Oct 29 23:07:56.504478 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.504286 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa0b407-b14b-47c2-a6d7-a16536017857-plugins-config-volume" pod "5fa0b407-b14b-47c2-a6d7-a16536017857" (UID: "5fa0b407-b14b-47c2-a6d7-a16536017857"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:56.505123 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.505083 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41347d77-4ec9-4e28-8651-6868480061c4-plugins-config-volume" pod "41347d77-4ec9-4e28-8651-6868480061c4" (UID: "41347d77-4ec9-4e28-8651-6868480061c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:56.506593 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.506573 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa0b407-b14b-47c2-a6d7-a16536017857-kube-api-access-m2v6f" pod "5fa0b407-b14b-47c2-a6d7-a16536017857" (UID: "5fa0b407-b14b-47c2-a6d7-a16536017857"). InnerVolumeSpecName "kube-api-access-m2v6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:56.506862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.506848 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41347d77-4ec9-4e28-8651-6868480061c4-kube-api-access-9tnxd" pod "41347d77-4ec9-4e28-8651-6868480061c4" (UID: "41347d77-4ec9-4e28-8651-6868480061c4"). InnerVolumeSpecName "kube-api-access-9tnxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:56.605600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.605566 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tnxd\" (UniqueName: \"kubernetes.io/projected/41347d77-4ec9-4e28-8651-6868480061c4-kube-api-access-9tnxd\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:56.605600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.605602 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fa0b407-b14b-47c2-a6d7-a16536017857-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:56.605806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.605618 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41347d77-4ec9-4e28-8651-6868480061c4-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:56.605806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:56.605631 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2v6f\" (UniqueName: \"kubernetes.io/projected/5fa0b407-b14b-47c2-a6d7-a16536017857-kube-api-access-m2v6f\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:07:57.415120 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:57.415090 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc" Oct 29 23:07:57.417079 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:57.417056 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8" Oct 29 23:07:57.417243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:57.417097 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8" event={"ID":"41347d77-4ec9-4e28-8651-6868480061c4","Type":"ContainerDied","Data":"2ec5962684d0c3ce2a5ec684dce6adbef4fde9b9bb20d52e80761db847674abf"} Oct 29 23:07:57.417243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:57.417157 5057 scope.go:122] "RemoveContainer" containerID="1d4c57ca2fb3f6789f3964c8b38d46e58225853e87355ff3575ddb7da50e8885" Oct 29 23:07:57.565366 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:57.565333 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc"] Oct 29 23:07:57.565817 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:57.565384 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0941c8-665846cc86-phzpc"] Oct 29 23:07:57.568652 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:57.568626 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8"] Oct 29 23:07:57.582172 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:57.582143 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0941c8-56856cb9d7-9zrh8"] Oct 29 23:07:59.103257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:59.103223 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t" Oct 29 23:07:59.104428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:59.104404 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc" Oct 29 23:07:59.538212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:59.537148 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41347d77-4ec9-4e28-8651-6868480061c4" path="/var/lib/kubelet/pods/41347d77-4ec9-4e28-8651-6868480061c4/volumes" Oct 29 23:07:59.538212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:07:59.537697 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fa0b407-b14b-47c2-a6d7-a16536017857" path="/var/lib/kubelet/pods/5fa0b407-b14b-47c2-a6d7-a16536017857/volumes" Oct 29 23:08:01.308102 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:01.307714 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": read tcp 10.132.0.2:55892->10.132.0.169:15021: read: connection reset by peer" start-of-body= Oct 29 23:08:01.308102 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:01.307784 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" podUID="8fd55749-31a3-49ff-8965-057fb2eeb9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": read tcp 10.132.0.2:55892->10.132.0.169:15021: read: connection reset by peer" Oct 29 23:08:01.557476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:01.557358 5057 generic.go:356] "Generic (PLEG): container finished" podID="8fd55749-31a3-49ff-8965-057fb2eeb9b8" containerID="7a3ed1f921a2cf7022e03834d6e2feba3cf62997b2a11a5f7c073025df0c75c7" exitCode=0 Oct 29 23:08:01.557476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:01.557435 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" event={"ID":"8fd55749-31a3-49ff-8965-057fb2eeb9b8","Type":"ContainerDied","Data":"7a3ed1f921a2cf7022e03834d6e2feba3cf62997b2a11a5f7c073025df0c75c7"} Oct 29 23:08:01.557943 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:01.557924 5057 scope.go:122] "RemoveContainer" containerID="7a3ed1f921a2cf7022e03834d6e2feba3cf62997b2a11a5f7c073025df0c75c7" Oct 29 23:08:01.560233 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:01.560180 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:08:01.560467 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:01.560438 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:08:01.560574 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:01.560500 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:08:02.572560 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:02.572364 5057 generic.go:356] "Generic (PLEG): container finished" podID="83a475a5-9dd5-4278-9109-97682e026b7c" containerID="02c4745abefb7e2bb3424c728f63190c38ab9a773651bc308637f4ed6bc02be1" exitCode=0 Oct 29 23:08:02.572560 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:02.572422 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" event={"ID":"83a475a5-9dd5-4278-9109-97682e026b7c","Type":"ContainerDied","Data":"02c4745abefb7e2bb3424c728f63190c38ab9a773651bc308637f4ed6bc02be1"} Oct 29 23:08:02.573199 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:02.572996 5057 scope.go:122] "RemoveContainer" containerID="02c4745abefb7e2bb3424c728f63190c38ab9a773651bc308637f4ed6bc02be1" Oct 29 23:08:02.575304 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:02.575262 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:08:02.575423 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:02.575356 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:08:02.575423 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:02.575411 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:08:02.577171 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:02.577151 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" event={"ID":"8fd55749-31a3-49ff-8965-057fb2eeb9b8","Type":"ContainerStarted","Data":"45e2ae4f8f200c2565f0f66877818df0ca5b7e12c08cdac04c600d6f45f52630"} Oct 29 23:08:02.793261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:02.789769 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:08:02.793261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:02.789953 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:08:03.588396 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:03.588360 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" event={"ID":"83a475a5-9dd5-4278-9109-97682e026b7c","Type":"ContainerStarted","Data":"16963e219c6944a6f0cb2c5f7dcff4c6c9e06b4235acd42af69cfa6cb6bbf3fa"} Oct 29 23:08:03.643565 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:03.641250 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:08:03.648213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:03.647360 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 29 23:08:03.648213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:03.647426 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" podUID="83a475a5-9dd5-4278-9109-97682e026b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 29 23:08:03.786795 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:03.786759 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 29 23:08:03.787072 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:03.786821 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" podUID="8fd55749-31a3-49ff-8965-057fb2eeb9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 29 23:08:03.919762 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:08:03.919672 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:08:04.650208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.648205 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 29 23:08:04.650208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.648320 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" podUID="83a475a5-9dd5-4278-9109-97682e026b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 29 23:08:04.700381 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.700343 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8"] Oct 29 23:08:04.732409 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.732376 5057 cpu_manager.go:414] "containerMap: removing container" podUID="41347d77-4ec9-4e28-8651-6868480061c4" containerName="payload-pre-processing" Oct 29 23:08:04.732656 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.732636 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41347d77-4ec9-4e28-8651-6868480061c4" containerName="payload-pre-processing" Oct 29 23:08:04.732794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.732779 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d728ed56-8f9a-4396-8386-7ce602570421" containerName="istio-proxy" Oct 29 23:08:04.732901 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.732889 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d728ed56-8f9a-4396-8386-7ce602570421" containerName="istio-proxy" Oct 29 23:08:04.733010 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.732999 5057 cpu_manager.go:414] "containerMap: removing container" podUID="927fbba6-8dbc-4e71-81d9-de5b2859ba5d" containerName="istio-proxy" Oct 29 23:08:04.733108 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.733097 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="927fbba6-8dbc-4e71-81d9-de5b2859ba5d" containerName="istio-proxy" Oct 29 23:08:04.733236 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.733223 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5fa0b407-b14b-47c2-a6d7-a16536017857" containerName="payload-processing" Oct 29 23:08:04.733368 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.733356 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa0b407-b14b-47c2-a6d7-a16536017857" containerName="payload-processing" Oct 29 23:08:04.733686 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.733629 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7d0539b1-a376-4016-900e-382ad30b3e02" containerName="istio-proxy" Oct 29 23:08:04.733910 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.733894 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d0539b1-a376-4016-900e-382ad30b3e02" containerName="istio-proxy" Oct 29 23:08:04.734507 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.734490 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="41347d77-4ec9-4e28-8651-6868480061c4" containerName="payload-pre-processing" Oct 29 23:08:04.734823 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.734808 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d728ed56-8f9a-4396-8386-7ce602570421" containerName="istio-proxy" Oct 29 23:08:04.734942 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.734931 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="927fbba6-8dbc-4e71-81d9-de5b2859ba5d" containerName="istio-proxy" Oct 29 23:08:04.735053 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.735042 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa0b407-b14b-47c2-a6d7-a16536017857" containerName="payload-processing" Oct 29 23:08:04.735238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.735225 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d0539b1-a376-4016-900e-382ad30b3e02" containerName="istio-proxy" Oct 29 23:08:04.802245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.802213 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8" Oct 29 23:08:04.850951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.841439 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-921680-dockercfg-xrv2x\"" Oct 29 23:08:04.850951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.841550 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-921680\"" Oct 29 23:08:04.850951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.843799 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 29 23:08:04.850951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.843867 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" podUID="8fd55749-31a3-49ff-8965-057fb2eeb9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 29 23:08:04.896548 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.896514 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8"] Oct 29 23:08:04.935271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.916390 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9w9b\" (UniqueName: \"kubernetes.io/projected/a6e31a55-feaa-4b94-bb1f-5f7758706537-kube-api-access-l9w9b\") pod \"payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8\" (UID: \"a6e31a55-feaa-4b94-bb1f-5f7758706537\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8" Oct 29 23:08:04.935271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:04.916708 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e31a55-feaa-4b94-bb1f-5f7758706537-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8\" (UID: \"a6e31a55-feaa-4b94-bb1f-5f7758706537\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8" Oct 29 23:08:05.006423 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.006376 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59"] Oct 29 23:08:05.008311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.008271 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d728ed56-8f9a-4396-8386-7ce602570421" containerName="istio-proxy" Oct 29 23:08:05.008495 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.008483 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d728ed56-8f9a-4396-8386-7ce602570421" containerName="istio-proxy" Oct 29 23:08:05.008836 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.008810 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d728ed56-8f9a-4396-8386-7ce602570421" containerName="istio-proxy" Oct 29 23:08:05.013475 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.013437 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59" Oct 29 23:08:05.015585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.015562 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59"] Oct 29 23:08:05.018023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.017992 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9w9b\" (UniqueName: \"kubernetes.io/projected/a6e31a55-feaa-4b94-bb1f-5f7758706537-kube-api-access-l9w9b\") pod \"payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8\" (UID: \"a6e31a55-feaa-4b94-bb1f-5f7758706537\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8" Oct 29 23:08:05.018424 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.018403 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e31a55-feaa-4b94-bb1f-5f7758706537-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8\" (UID: \"a6e31a55-feaa-4b94-bb1f-5f7758706537\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8" Oct 29 23:08:05.019750 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.019725 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e31a55-feaa-4b94-bb1f-5f7758706537-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8\" (UID: \"a6e31a55-feaa-4b94-bb1f-5f7758706537\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8" Oct 29 23:08:05.084295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.084264 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9w9b\" (UniqueName: \"kubernetes.io/projected/a6e31a55-feaa-4b94-bb1f-5f7758706537-kube-api-access-l9w9b\") pod \"payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8\" (UID: \"a6e31a55-feaa-4b94-bb1f-5f7758706537\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8" Oct 29 23:08:05.119764 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.119731 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884eed63-5693-41b6-81c6-0d4cd1abdc4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59\" (UID: \"884eed63-5693-41b6-81c6-0d4cd1abdc4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59" Oct 29 23:08:05.119955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.119830 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v72kr\" (UniqueName: \"kubernetes.io/projected/884eed63-5693-41b6-81c6-0d4cd1abdc4c-kube-api-access-v72kr\") pod \"payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59\" (UID: \"884eed63-5693-41b6-81c6-0d4cd1abdc4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59" Oct 29 23:08:05.229265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.222126 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884eed63-5693-41b6-81c6-0d4cd1abdc4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59\" (UID: \"884eed63-5693-41b6-81c6-0d4cd1abdc4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59" Oct 29 23:08:05.229265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.222208 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v72kr\" (UniqueName: \"kubernetes.io/projected/884eed63-5693-41b6-81c6-0d4cd1abdc4c-kube-api-access-v72kr\") pod \"payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59\" (UID: \"884eed63-5693-41b6-81c6-0d4cd1abdc4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59" Oct 29 23:08:05.229265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.223605 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884eed63-5693-41b6-81c6-0d4cd1abdc4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59\" (UID: \"884eed63-5693-41b6-81c6-0d4cd1abdc4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59" Oct 29 23:08:05.255434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.255402 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v72kr\" (UniqueName: \"kubernetes.io/projected/884eed63-5693-41b6-81c6-0d4cd1abdc4c-kube-api-access-v72kr\") pod \"payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59\" (UID: \"884eed63-5693-41b6-81c6-0d4cd1abdc4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59" Oct 29 23:08:05.319737 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.319704 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8" Oct 29 23:08:05.392786 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.392452 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59" Oct 29 23:08:05.804263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.788705 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 29 23:08:05.804263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.788772 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" podUID="83a475a5-9dd5-4278-9109-97682e026b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 29 23:08:05.861206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.851200 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 29 23:08:05.861206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:05.851263 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" podUID="8fd55749-31a3-49ff-8965-057fb2eeb9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 29 23:08:06.200959 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:06.200871 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8"] Oct 29 23:08:06.296417 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:06.296369 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59"] Oct 29 23:08:06.302962 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:08:06.302929 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884eed63_5693_41b6_81c6_0d4cd1abdc4c.slice/crio-f15e0a3f10866757acec372cb559f5700b080b207779ded47be9d0b53d67ca53 WatchSource:0}: Error finding container f15e0a3f10866757acec372cb559f5700b080b207779ded47be9d0b53d67ca53: Status 404 returned error can't find the container with id f15e0a3f10866757acec372cb559f5700b080b207779ded47be9d0b53d67ca53 Oct 29 23:08:06.638990 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:06.638550 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 29 23:08:06.638990 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:06.638609 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" podUID="83a475a5-9dd5-4278-9109-97682e026b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 29 23:08:06.800740 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:06.800653 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 29 23:08:06.801418 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:06.800960 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" podUID="8fd55749-31a3-49ff-8965-057fb2eeb9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 29 23:08:07.017217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:07.015306 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59" event={"ID":"884eed63-5693-41b6-81c6-0d4cd1abdc4c","Type":"ContainerStarted","Data":"3d9145caff7e04764bf000d51ca9e9edc181b8d03e584c78bdf48135ab4cc0d6"} Oct 29 23:08:07.017217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:07.015360 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59" event={"ID":"884eed63-5693-41b6-81c6-0d4cd1abdc4c","Type":"ContainerStarted","Data":"f15e0a3f10866757acec372cb559f5700b080b207779ded47be9d0b53d67ca53"} Oct 29 23:08:07.017217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:07.016214 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59" Oct 29 23:08:07.017818 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:07.017736 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8" event={"ID":"a6e31a55-feaa-4b94-bb1f-5f7758706537","Type":"ContainerStarted","Data":"261570d035c22aad0faeadebd21a01b15bef2728d4dd09647c621b1709263776"} Oct 29 23:08:07.017818 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:07.017778 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8" event={"ID":"a6e31a55-feaa-4b94-bb1f-5f7758706537","Type":"ContainerStarted","Data":"88864bda60277be3daa19df843c7f624badf5bdb40f81853a4bb395eb25e30a3"} Oct 29 23:08:07.017818 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:07.017798 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8" Oct 29 23:08:07.072265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:07.071961 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59" podStartSLOduration=3.071929203 podStartE2EDuration="3.071929203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:07.07040745 +0000 UTC m=+1533.721905982" watchObservedRunningTime="2026-10-29 23:08:07.071929203 +0000 UTC m=+1533.723427733" Oct 29 23:08:07.122211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:07.119148 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8" podStartSLOduration=3.119129383 podStartE2EDuration="3.119129383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:07.118299694 +0000 UTC m=+1533.769798224" watchObservedRunningTime="2026-10-29 23:08:07.119129383 +0000 UTC m=+1533.770627920" Oct 29 23:08:07.677041 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:07.672166 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 29 23:08:07.677041 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:07.672254 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" podUID="83a475a5-9dd5-4278-9109-97682e026b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 29 23:08:07.788307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:07.788275 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 29 23:08:07.788575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:07.788544 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" podUID="8fd55749-31a3-49ff-8965-057fb2eeb9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 29 23:08:08.645208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:08.641632 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 29 23:08:08.645208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:08.641716 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" podUID="83a475a5-9dd5-4278-9109-97682e026b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 29 23:08:08.788211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:08.787644 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 29 23:08:08.788211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:08.787706 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" podUID="8fd55749-31a3-49ff-8965-057fb2eeb9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 29 23:08:09.578353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:09.578318 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t"] Oct 29 23:08:09.578835 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:09.578801 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" podUID="06c26371-4cdf-4bf1-95d7-855d452655c3" containerName="istio-proxy" containerID="cri-o://c5745238b5490434784905f117394c19ec0f52ac9c64389581fb85bf55fb72fd" gracePeriod=30 Oct 29 23:08:09.701567 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:09.701527 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 29 23:08:09.705348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:09.705302 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" podUID="83a475a5-9dd5-4278-9109-97682e026b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 29 23:08:10.372535 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:10.372501 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:08:10.374415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:10.374392 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:08:10.384912 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:10.384676 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:08:10.650256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:10.644621 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 29 23:08:10.650256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:10.644681 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" podUID="83a475a5-9dd5-4278-9109-97682e026b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 29 23:08:11.643967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:11.643934 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:08:11.644488 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:11.644474 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:08:11.645101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:11.645076 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:08:14.965155 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:14.965125 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:08:15.073846 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.073798 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c26371-4cdf-4bf1-95d7-855d452655c3-kube-api-access-78p4d\" (UniqueName: \"kubernetes.io/projected/06c26371-4cdf-4bf1-95d7-855d452655c3-kube-api-access-78p4d\") pod \"06c26371-4cdf-4bf1-95d7-855d452655c3\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " Oct 29 23:08:15.074019 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.073907 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06c26371-4cdf-4bf1-95d7-855d452655c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c26371-4cdf-4bf1-95d7-855d452655c3-istiod-ca-cert\") pod \"06c26371-4cdf-4bf1-95d7-855d452655c3\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " Oct 29 23:08:15.074019 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.073985 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-workload-certs\") pod \"06c26371-4cdf-4bf1-95d7-855d452655c3\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " Oct 29 23:08:15.074098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.074041 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-data\") pod \"06c26371-4cdf-4bf1-95d7-855d452655c3\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " Oct 29 23:08:15.074098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.074092 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-credential-socket\") pod \"06c26371-4cdf-4bf1-95d7-855d452655c3\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " Oct 29 23:08:15.074182 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.074167 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-token\" (UniqueName: \"kubernetes.io/projected/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-token\") pod \"06c26371-4cdf-4bf1-95d7-855d452655c3\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " Oct 29 23:08:15.074309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.074223 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-workload-socket\") pod \"06c26371-4cdf-4bf1-95d7-855d452655c3\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " Oct 29 23:08:15.074309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.074289 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-envoy\") pod \"06c26371-4cdf-4bf1-95d7-855d452655c3\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " Oct 29 23:08:15.074373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.074319 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-podinfo\") pod \"06c26371-4cdf-4bf1-95d7-855d452655c3\" (UID: \"06c26371-4cdf-4bf1-95d7-855d452655c3\") " Oct 29 23:08:15.075639 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.075605 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-workload-socket" pod "06c26371-4cdf-4bf1-95d7-855d452655c3" (UID: "06c26371-4cdf-4bf1-95d7-855d452655c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:15.075769 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.075694 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c26371-4cdf-4bf1-95d7-855d452655c3-istiod-ca-cert" pod "06c26371-4cdf-4bf1-95d7-855d452655c3" (UID: "06c26371-4cdf-4bf1-95d7-855d452655c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:15.075840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.075798 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-workload-certs" pod "06c26371-4cdf-4bf1-95d7-855d452655c3" (UID: "06c26371-4cdf-4bf1-95d7-855d452655c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:15.076060 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.076034 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-data" pod "06c26371-4cdf-4bf1-95d7-855d452655c3" (UID: "06c26371-4cdf-4bf1-95d7-855d452655c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:15.076265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.076066 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-credential-socket" pod "06c26371-4cdf-4bf1-95d7-855d452655c3" (UID: "06c26371-4cdf-4bf1-95d7-855d452655c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:15.078331 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.078306 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-envoy" pod "06c26371-4cdf-4bf1-95d7-855d452655c3" (UID: "06c26371-4cdf-4bf1-95d7-855d452655c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:15.078441 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.078361 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-token" pod "06c26371-4cdf-4bf1-95d7-855d452655c3" (UID: "06c26371-4cdf-4bf1-95d7-855d452655c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:15.078441 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.078383 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c26371-4cdf-4bf1-95d7-855d452655c3-kube-api-access-78p4d" pod "06c26371-4cdf-4bf1-95d7-855d452655c3" (UID: "06c26371-4cdf-4bf1-95d7-855d452655c3"). InnerVolumeSpecName "kube-api-access-78p4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:15.079317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.079299 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-podinfo" pod "06c26371-4cdf-4bf1-95d7-855d452655c3" (UID: "06c26371-4cdf-4bf1-95d7-855d452655c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:08:15.160647 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.160615 5057 generic.go:356] "Generic (PLEG): container finished" podID="06c26371-4cdf-4bf1-95d7-855d452655c3" containerID="c5745238b5490434784905f117394c19ec0f52ac9c64389581fb85bf55fb72fd" exitCode=0 Oct 29 23:08:15.160820 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.160680 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" Oct 29 23:08:15.160820 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.160716 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" event={"ID":"06c26371-4cdf-4bf1-95d7-855d452655c3","Type":"ContainerDied","Data":"c5745238b5490434784905f117394c19ec0f52ac9c64389581fb85bf55fb72fd"} Oct 29 23:08:15.160820 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.160785 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t" event={"ID":"06c26371-4cdf-4bf1-95d7-855d452655c3","Type":"ContainerDied","Data":"bed97bc4b3b29056db0312884d6b3a9e20c4b27844aa598d890c4cbb9a64f7ce"} Oct 29 23:08:15.160820 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.160814 5057 scope.go:122] "RemoveContainer" containerID="c5745238b5490434784905f117394c19ec0f52ac9c64389581fb85bf55fb72fd" Oct 29 23:08:15.173568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.173549 5057 scope.go:122] "RemoveContainer" containerID="95de34c0859f5b7e5cbf48288a72e2d852ca439d02b673d5b3619997dec7168e" Oct 29 23:08:15.177746 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.177721 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:15.177834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.177750 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:15.177834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.177763 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:15.177834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.177774 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:15.177834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.177784 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:15.177834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.177792 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78p4d\" (UniqueName: \"kubernetes.io/projected/06c26371-4cdf-4bf1-95d7-855d452655c3-kube-api-access-78p4d\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:15.177834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.177802 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c26371-4cdf-4bf1-95d7-855d452655c3-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:15.177834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.177810 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:15.177834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.177821 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c26371-4cdf-4bf1-95d7-855d452655c3-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:15.183801 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.183779 5057 scope.go:122] "RemoveContainer" containerID="c5745238b5490434784905f117394c19ec0f52ac9c64389581fb85bf55fb72fd" Oct 29 23:08:15.184069 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:08:15.184042 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5745238b5490434784905f117394c19ec0f52ac9c64389581fb85bf55fb72fd\": container with ID starting with c5745238b5490434784905f117394c19ec0f52ac9c64389581fb85bf55fb72fd not found: ID does not exist" containerID="c5745238b5490434784905f117394c19ec0f52ac9c64389581fb85bf55fb72fd" Oct 29 23:08:15.184152 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.184081 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5745238b5490434784905f117394c19ec0f52ac9c64389581fb85bf55fb72fd"} err="failed to get container status \"c5745238b5490434784905f117394c19ec0f52ac9c64389581fb85bf55fb72fd\": rpc error: code = NotFound desc = could not find container \"c5745238b5490434784905f117394c19ec0f52ac9c64389581fb85bf55fb72fd\": container with ID starting with c5745238b5490434784905f117394c19ec0f52ac9c64389581fb85bf55fb72fd not found: ID does not exist" Oct 29 23:08:15.184152 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.184105 5057 scope.go:122] "RemoveContainer" containerID="95de34c0859f5b7e5cbf48288a72e2d852ca439d02b673d5b3619997dec7168e" Oct 29 23:08:15.184400 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:08:15.184381 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95de34c0859f5b7e5cbf48288a72e2d852ca439d02b673d5b3619997dec7168e\": container with ID starting with 95de34c0859f5b7e5cbf48288a72e2d852ca439d02b673d5b3619997dec7168e not found: ID does not exist" containerID="95de34c0859f5b7e5cbf48288a72e2d852ca439d02b673d5b3619997dec7168e" Oct 29 23:08:15.184447 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.184406 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95de34c0859f5b7e5cbf48288a72e2d852ca439d02b673d5b3619997dec7168e"} err="failed to get container status \"95de34c0859f5b7e5cbf48288a72e2d852ca439d02b673d5b3619997dec7168e\": rpc error: code = NotFound desc = could not find container \"95de34c0859f5b7e5cbf48288a72e2d852ca439d02b673d5b3619997dec7168e\": container with ID starting with 95de34c0859f5b7e5cbf48288a72e2d852ca439d02b673d5b3619997dec7168e not found: ID does not exist" Oct 29 23:08:15.220988 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.220958 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t"] Oct 29 23:08:15.225878 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.225851 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-0941c8-openshift-default-757f85dc9c-nbf4t"] Oct 29 23:08:15.483121 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:15.483078 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06c26371-4cdf-4bf1-95d7-855d452655c3" path="/var/lib/kubelet/pods/06c26371-4cdf-4bf1-95d7-855d452655c3/volumes" Oct 29 23:08:18.072920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:18.072891 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8" Oct 29 23:08:19.104981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:19.104952 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59" Oct 29 23:08:20.666211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:20.663071 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7"] Oct 29 23:08:21.335216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:21.332920 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4"] Oct 29 23:08:22.452226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.452173 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762"] Oct 29 23:08:22.453365 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.453343 5057 cpu_manager.go:414] "containerMap: removing container" podUID="06c26371-4cdf-4bf1-95d7-855d452655c3" containerName="istio-proxy" Oct 29 23:08:22.453434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.453366 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c26371-4cdf-4bf1-95d7-855d452655c3" containerName="istio-proxy" Oct 29 23:08:22.453434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.453393 5057 cpu_manager.go:414] "containerMap: removing container" podUID="06c26371-4cdf-4bf1-95d7-855d452655c3" containerName="istio-proxy" Oct 29 23:08:22.453434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.453401 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c26371-4cdf-4bf1-95d7-855d452655c3" containerName="istio-proxy" Oct 29 23:08:22.453646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.453634 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c26371-4cdf-4bf1-95d7-855d452655c3" containerName="istio-proxy" Oct 29 23:08:22.453681 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.453652 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c26371-4cdf-4bf1-95d7-855d452655c3" containerName="istio-proxy" Oct 29 23:08:22.459879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.459856 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" Oct 29 23:08:22.481051 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.481021 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762"] Oct 29 23:08:22.485828 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.485802 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-6d21fd-serving-cert\"" Oct 29 23:08:22.578578 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.578542 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-metrics-certs\") pod \"maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762\" (UID: \"b384d2c7-ccaa-4655-a85a-2d1c16fc178a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" Oct 29 23:08:22.578767 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.578754 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762\" (UID: \"b384d2c7-ccaa-4655-a85a-2d1c16fc178a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" Oct 29 23:08:22.578819 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.578796 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762\" (UID: \"b384d2c7-ccaa-4655-a85a-2d1c16fc178a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" Oct 29 23:08:22.578897 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.578883 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcjjs\" (UniqueName: \"kubernetes.io/projected/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-kube-api-access-rcjjs\") pod \"maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762\" (UID: \"b384d2c7-ccaa-4655-a85a-2d1c16fc178a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" Oct 29 23:08:22.684448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.684399 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcjjs\" (UniqueName: \"kubernetes.io/projected/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-kube-api-access-rcjjs\") pod \"maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762\" (UID: \"b384d2c7-ccaa-4655-a85a-2d1c16fc178a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" Oct 29 23:08:22.684690 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.684594 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-metrics-certs\") pod \"maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762\" (UID: \"b384d2c7-ccaa-4655-a85a-2d1c16fc178a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" Oct 29 23:08:22.684813 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.684792 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762\" (UID: \"b384d2c7-ccaa-4655-a85a-2d1c16fc178a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" Oct 29 23:08:22.684890 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.684859 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762\" (UID: \"b384d2c7-ccaa-4655-a85a-2d1c16fc178a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" Oct 29 23:08:22.692211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.687809 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762\" (UID: \"b384d2c7-ccaa-4655-a85a-2d1c16fc178a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" Oct 29 23:08:22.725295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.722097 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-metrics-certs\") pod \"maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762\" (UID: \"b384d2c7-ccaa-4655-a85a-2d1c16fc178a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" Oct 29 23:08:22.735238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.726393 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762\" (UID: \"b384d2c7-ccaa-4655-a85a-2d1c16fc178a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" Oct 29 23:08:22.735238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.729528 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcjjs\" (UniqueName: \"kubernetes.io/projected/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-kube-api-access-rcjjs\") pod \"maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762\" (UID: \"b384d2c7-ccaa-4655-a85a-2d1c16fc178a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" Oct 29 23:08:22.789247 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:22.779487 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" Oct 29 23:08:23.138396 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:08:23.138367 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb384d2c7_ccaa_4655_a85a_2d1c16fc178a.slice/crio-0097279e8fff87415496dad249cbd2a576fa345bda0a086af684717f5d83b72c WatchSource:0}: Error finding container 0097279e8fff87415496dad249cbd2a576fa345bda0a086af684717f5d83b72c: Status 404 returned error can't find the container with id 0097279e8fff87415496dad249cbd2a576fa345bda0a086af684717f5d83b72c Oct 29 23:08:23.140844 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:23.140820 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762"] Oct 29 23:08:23.279331 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:23.279298 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" event={"ID":"b384d2c7-ccaa-4655-a85a-2d1c16fc178a","Type":"ContainerStarted","Data":"0097279e8fff87415496dad249cbd2a576fa345bda0a086af684717f5d83b72c"} Oct 29 23:08:24.286513 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.286481 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh"] Oct 29 23:08:24.291317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.291292 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.291986 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.291960 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" event={"ID":"b384d2c7-ccaa-4655-a85a-2d1c16fc178a","Type":"ContainerStarted","Data":"5c2c2fb1712bbdbfc43dc5ff17ed2fae7fd1f786ecdb78e6fcfc8cd2786192af"} Oct 29 23:08:24.292176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.292150 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" Oct 29 23:08:24.308788 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.308637 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-31d798-openshift-default-dockercfg-t78cb\"" Oct 29 23:08:24.345209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.333859 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-envoy\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.345209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.333939 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btpd7\" (UniqueName: \"kubernetes.io/projected/d35fd058-ef20-4d71-bd81-0ecf36bef728-kube-api-access-btpd7\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.345209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.333973 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-workload-socket\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.345209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.333999 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-credential-socket\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.345209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.334090 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-workload-certs\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.345209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.334255 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-podinfo\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.345209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.334298 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d35fd058-ef20-4d71-bd81-0ecf36bef728-istiod-ca-cert\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.345209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.334329 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-data\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.345209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.334373 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-token\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.345209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.338029 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh"] Oct 29 23:08:24.444209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.443757 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-token\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.448860 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.448792 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" podStartSLOduration=2.02098614 podStartE2EDuration="2.448771949s" totalImagesPullingTime="427.785809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:24.394366077 +0000 UTC m=+1551.045864606" watchObservedRunningTime="2026-10-29 23:08:24.448771949 +0000 UTC m=+1551.100270484" Oct 29 23:08:24.478234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.470168 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-envoy\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.478234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.470292 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btpd7\" (UniqueName: \"kubernetes.io/projected/d35fd058-ef20-4d71-bd81-0ecf36bef728-kube-api-access-btpd7\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.478234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.470327 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-workload-socket\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.478234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.470352 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-credential-socket\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.478234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.470481 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-workload-certs\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.478234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.470651 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-podinfo\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.478234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.470683 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d35fd058-ef20-4d71-bd81-0ecf36bef728-istiod-ca-cert\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.478234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.470714 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-data\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.478234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.471219 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-data\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.493249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.479250 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-credential-socket\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.493249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.479530 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-workload-socket\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.493249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.479764 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-workload-certs\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.493249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.480390 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d35fd058-ef20-4d71-bd81-0ecf36bef728-istiod-ca-cert\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.538151 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.538072 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-token\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.592227 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.552977 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btpd7\" (UniqueName: \"kubernetes.io/projected/d35fd058-ef20-4d71-bd81-0ecf36bef728-kube-api-access-btpd7\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.614221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.601561 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-envoy\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.614221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.606775 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-podinfo\") pod \"e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:24.614221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:24.613472 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:25.099905 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:25.095161 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh"] Oct 29 23:08:25.105490 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:08:25.105456 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35fd058_ef20_4d71_bd81_0ecf36bef728.slice/crio-fe9c171cafa6ce4449505b58a0962769365a955a9071b4aacbc6057f5d83c22a WatchSource:0}: Error finding container fe9c171cafa6ce4449505b58a0962769365a955a9071b4aacbc6057f5d83c22a: Status 404 returned error can't find the container with id fe9c171cafa6ce4449505b58a0962769365a955a9071b4aacbc6057f5d83c22a Oct 29 23:08:25.108667 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:25.108633 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:08:25.108807 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:25.108701 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:08:25.108807 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:25.108729 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:08:25.303749 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:25.303705 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" event={"ID":"d35fd058-ef20-4d71-bd81-0ecf36bef728","Type":"ContainerStarted","Data":"9b043a9d42fbfc35ec3d5d792e10dc334e5bd73f1070aa14c95ff1e197eacbaf"} Oct 29 23:08:25.304143 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:25.303787 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" event={"ID":"d35fd058-ef20-4d71-bd81-0ecf36bef728","Type":"ContainerStarted","Data":"fe9c171cafa6ce4449505b58a0962769365a955a9071b4aacbc6057f5d83c22a"} Oct 29 23:08:25.356470 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:25.356357 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" podStartSLOduration=1.356340159 podStartE2EDuration="1.356340159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:25.351597423 +0000 UTC m=+1552.003095956" watchObservedRunningTime="2026-10-29 23:08:25.356340159 +0000 UTC m=+1552.007838689" Oct 29 23:08:25.615331 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:25.615242 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:25.663542 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:25.663484 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7" podUID="7765cde8-749b-4360-858b-b56100c5037d" containerName="payload-pre-processing" containerID="cri-o://854d30196b12d63b94d373e2980945d78281dc70c35a72e41886af8c2db0a7a9" gracePeriod=25 Oct 29 23:08:25.969143 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:25.969114 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7" Oct 29 23:08:26.017772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.017737 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7765cde8-749b-4360-858b-b56100c5037d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7765cde8-749b-4360-858b-b56100c5037d-plugins-config-volume\") pod \"7765cde8-749b-4360-858b-b56100c5037d\" (UID: \"7765cde8-749b-4360-858b-b56100c5037d\") " Oct 29 23:08:26.018023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.018005 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7765cde8-749b-4360-858b-b56100c5037d-kube-api-access-sghzx\" (UniqueName: \"kubernetes.io/projected/7765cde8-749b-4360-858b-b56100c5037d-kube-api-access-sghzx\") pod \"7765cde8-749b-4360-858b-b56100c5037d\" (UID: \"7765cde8-749b-4360-858b-b56100c5037d\") " Oct 29 23:08:26.018241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.018214 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7765cde8-749b-4360-858b-b56100c5037d-plugins-config-volume" pod "7765cde8-749b-4360-858b-b56100c5037d" (UID: "7765cde8-749b-4360-858b-b56100c5037d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:26.018771 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.018745 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7765cde8-749b-4360-858b-b56100c5037d-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:26.020594 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.020570 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7765cde8-749b-4360-858b-b56100c5037d-kube-api-access-sghzx" pod "7765cde8-749b-4360-858b-b56100c5037d" (UID: "7765cde8-749b-4360-858b-b56100c5037d"). InnerVolumeSpecName "kube-api-access-sghzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:26.120348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.120310 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sghzx\" (UniqueName: \"kubernetes.io/projected/7765cde8-749b-4360-858b-b56100c5037d-kube-api-access-sghzx\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:26.313148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.313114 5057 generic.go:356] "Generic (PLEG): container finished" podID="7765cde8-749b-4360-858b-b56100c5037d" containerID="854d30196b12d63b94d373e2980945d78281dc70c35a72e41886af8c2db0a7a9" exitCode=0 Oct 29 23:08:26.313581 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.313162 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7" Oct 29 23:08:26.313581 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.313233 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7" event={"ID":"7765cde8-749b-4360-858b-b56100c5037d","Type":"ContainerDied","Data":"854d30196b12d63b94d373e2980945d78281dc70c35a72e41886af8c2db0a7a9"} Oct 29 23:08:26.313581 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.313287 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7" event={"ID":"7765cde8-749b-4360-858b-b56100c5037d","Type":"ContainerDied","Data":"230e185f7baaac7c9e50bbb9552df9fea9d8510c6686c2bf702da3b60b8fa03b"} Oct 29 23:08:26.313581 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.313316 5057 scope.go:122] "RemoveContainer" containerID="854d30196b12d63b94d373e2980945d78281dc70c35a72e41886af8c2db0a7a9" Oct 29 23:08:26.326611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.326590 5057 scope.go:122] "RemoveContainer" containerID="854d30196b12d63b94d373e2980945d78281dc70c35a72e41886af8c2db0a7a9" Oct 29 23:08:26.326899 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:08:26.326880 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854d30196b12d63b94d373e2980945d78281dc70c35a72e41886af8c2db0a7a9\": container with ID starting with 854d30196b12d63b94d373e2980945d78281dc70c35a72e41886af8c2db0a7a9 not found: ID does not exist" containerID="854d30196b12d63b94d373e2980945d78281dc70c35a72e41886af8c2db0a7a9" Oct 29 23:08:26.326984 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.326909 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854d30196b12d63b94d373e2980945d78281dc70c35a72e41886af8c2db0a7a9"} err="failed to get container status \"854d30196b12d63b94d373e2980945d78281dc70c35a72e41886af8c2db0a7a9\": rpc error: code = NotFound desc = could not find container \"854d30196b12d63b94d373e2980945d78281dc70c35a72e41886af8c2db0a7a9\": container with ID starting with 854d30196b12d63b94d373e2980945d78281dc70c35a72e41886af8c2db0a7a9 not found: ID does not exist" Oct 29 23:08:26.334124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.334056 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4" podUID="dd3473cd-60a2-4f67-9da9-6ed27a733f44" containerName="payload-pre-processing" containerID="cri-o://985ffd8ded60a0345bafce53d688f14b83f38d9e5fda113021c4a19f7bf4d7c9" gracePeriod=25 Oct 29 23:08:26.372153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.372127 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7"] Oct 29 23:08:26.386001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.385975 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-957eef-5ff86dfc67-6cmt7"] Oct 29 23:08:26.622365 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.622333 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:26.631344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.631167 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4" Oct 29 23:08:26.738036 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.737600 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd3473cd-60a2-4f67-9da9-6ed27a733f44-kube-api-access-jnlk9\" (UniqueName: \"kubernetes.io/projected/dd3473cd-60a2-4f67-9da9-6ed27a733f44-kube-api-access-jnlk9\") pod \"dd3473cd-60a2-4f67-9da9-6ed27a733f44\" (UID: \"dd3473cd-60a2-4f67-9da9-6ed27a733f44\") " Oct 29 23:08:26.738036 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.737684 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd3473cd-60a2-4f67-9da9-6ed27a733f44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3473cd-60a2-4f67-9da9-6ed27a733f44-plugins-config-volume\") pod \"dd3473cd-60a2-4f67-9da9-6ed27a733f44\" (UID: \"dd3473cd-60a2-4f67-9da9-6ed27a733f44\") " Oct 29 23:08:26.738508 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.738480 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3473cd-60a2-4f67-9da9-6ed27a733f44-plugins-config-volume" pod "dd3473cd-60a2-4f67-9da9-6ed27a733f44" (UID: "dd3473cd-60a2-4f67-9da9-6ed27a733f44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:26.739452 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.739413 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3473cd-60a2-4f67-9da9-6ed27a733f44-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:26.740665 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.740641 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3473cd-60a2-4f67-9da9-6ed27a733f44-kube-api-access-jnlk9" pod "dd3473cd-60a2-4f67-9da9-6ed27a733f44" (UID: "dd3473cd-60a2-4f67-9da9-6ed27a733f44"). InnerVolumeSpecName "kube-api-access-jnlk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:26.910259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:26.905336 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnlk9\" (UniqueName: \"kubernetes.io/projected/dd3473cd-60a2-4f67-9da9-6ed27a733f44-kube-api-access-jnlk9\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:27.324451 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:27.324413 5057 generic.go:356] "Generic (PLEG): container finished" podID="dd3473cd-60a2-4f67-9da9-6ed27a733f44" containerID="985ffd8ded60a0345bafce53d688f14b83f38d9e5fda113021c4a19f7bf4d7c9" exitCode=0 Oct 29 23:08:27.324880 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:27.324453 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4" Oct 29 23:08:27.324880 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:27.324501 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4" event={"ID":"dd3473cd-60a2-4f67-9da9-6ed27a733f44","Type":"ContainerDied","Data":"985ffd8ded60a0345bafce53d688f14b83f38d9e5fda113021c4a19f7bf4d7c9"} Oct 29 23:08:27.324880 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:27.324574 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4" event={"ID":"dd3473cd-60a2-4f67-9da9-6ed27a733f44","Type":"ContainerDied","Data":"72f44c60f8dc4b065643c17ad1f6f50053deba8f638fd7592d9ceeeda194b3d5"} Oct 29 23:08:27.324880 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:27.324614 5057 scope.go:122] "RemoveContainer" containerID="985ffd8ded60a0345bafce53d688f14b83f38d9e5fda113021c4a19f7bf4d7c9" Oct 29 23:08:27.325212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:27.325160 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:27.327446 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:27.327423 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:08:27.348578 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:27.348550 5057 scope.go:122] "RemoveContainer" containerID="985ffd8ded60a0345bafce53d688f14b83f38d9e5fda113021c4a19f7bf4d7c9" Oct 29 23:08:27.348916 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:08:27.348892 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"985ffd8ded60a0345bafce53d688f14b83f38d9e5fda113021c4a19f7bf4d7c9\": container with ID starting with 985ffd8ded60a0345bafce53d688f14b83f38d9e5fda113021c4a19f7bf4d7c9 not found: ID does not exist" containerID="985ffd8ded60a0345bafce53d688f14b83f38d9e5fda113021c4a19f7bf4d7c9" Oct 29 23:08:27.349014 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:27.348926 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"985ffd8ded60a0345bafce53d688f14b83f38d9e5fda113021c4a19f7bf4d7c9"} err="failed to get container status \"985ffd8ded60a0345bafce53d688f14b83f38d9e5fda113021c4a19f7bf4d7c9\": rpc error: code = NotFound desc = could not find container \"985ffd8ded60a0345bafce53d688f14b83f38d9e5fda113021c4a19f7bf4d7c9\": container with ID starting with 985ffd8ded60a0345bafce53d688f14b83f38d9e5fda113021c4a19f7bf4d7c9 not found: ID does not exist" Oct 29 23:08:27.450503 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:27.450467 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4"] Oct 29 23:08:27.450742 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:27.450728 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ce5978-cbfdd6975-f8nh4"] Oct 29 23:08:27.486228 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:27.486180 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7765cde8-749b-4360-858b-b56100c5037d" path="/var/lib/kubelet/pods/7765cde8-749b-4360-858b-b56100c5037d/volumes" Oct 29 23:08:27.486773 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:27.486753 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd3473cd-60a2-4f67-9da9-6ed27a733f44" path="/var/lib/kubelet/pods/dd3473cd-60a2-4f67-9da9-6ed27a733f44/volumes" Oct 29 23:08:27.939124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:27.939090 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5"] Oct 29 23:08:29.674854 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:29.674816 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n"] Oct 29 23:08:29.675597 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:29.675582 5057 cpu_manager.go:414] "containerMap: removing container" podUID="dd3473cd-60a2-4f67-9da9-6ed27a733f44" containerName="payload-pre-processing" Oct 29 23:08:29.675645 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:29.675597 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3473cd-60a2-4f67-9da9-6ed27a733f44" containerName="payload-pre-processing" Oct 29 23:08:29.675645 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:29.675640 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7765cde8-749b-4360-858b-b56100c5037d" containerName="payload-pre-processing" Oct 29 23:08:29.675709 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:29.675647 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7765cde8-749b-4360-858b-b56100c5037d" containerName="payload-pre-processing" Oct 29 23:08:29.675767 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:29.675757 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7765cde8-749b-4360-858b-b56100c5037d" containerName="payload-pre-processing" Oct 29 23:08:29.675805 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:29.675773 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3473cd-60a2-4f67-9da9-6ed27a733f44" containerName="payload-pre-processing" Oct 29 23:08:29.679200 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:29.679170 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n" Oct 29 23:08:29.700291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:29.700262 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n"] Oct 29 23:08:29.708835 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:29.708800 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-31d798\"" Oct 29 23:08:29.709411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:29.709391 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-31d798-dockercfg-g6f69\"" Oct 29 23:08:29.744405 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:29.744364 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x7kw\" (UniqueName: \"kubernetes.io/projected/74819147-4b47-42d4-8d03-c12347c309bb-kube-api-access-2x7kw\") pod \"payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n\" (UID: \"74819147-4b47-42d4-8d03-c12347c309bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n" Oct 29 23:08:29.744637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:29.744522 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74819147-4b47-42d4-8d03-c12347c309bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n\" (UID: \"74819147-4b47-42d4-8d03-c12347c309bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n" Oct 29 23:08:29.892214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:29.891848 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x7kw\" (UniqueName: \"kubernetes.io/projected/74819147-4b47-42d4-8d03-c12347c309bb-kube-api-access-2x7kw\") pod \"payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n\" (UID: \"74819147-4b47-42d4-8d03-c12347c309bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n" Oct 29 23:08:29.892214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:29.891985 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74819147-4b47-42d4-8d03-c12347c309bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n\" (UID: \"74819147-4b47-42d4-8d03-c12347c309bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n" Oct 29 23:08:29.926262 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:29.920127 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74819147-4b47-42d4-8d03-c12347c309bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n\" (UID: \"74819147-4b47-42d4-8d03-c12347c309bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n" Oct 29 23:08:29.952205 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:29.948301 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x7kw\" (UniqueName: \"kubernetes.io/projected/74819147-4b47-42d4-8d03-c12347c309bb-kube-api-access-2x7kw\") pod \"payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n\" (UID: \"74819147-4b47-42d4-8d03-c12347c309bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n" Oct 29 23:08:30.008213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:29.998662 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n" Oct 29 23:08:30.309995 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:30.309965 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" Oct 29 23:08:30.456593 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:08:30.456519 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74819147_4b47_42d4_8d03_c12347c309bb.slice/crio-cb857a29f11bec60cada3302e945c7c0accb94078ef7412aedf62e100c42dcc8 WatchSource:0}: Error finding container cb857a29f11bec60cada3302e945c7c0accb94078ef7412aedf62e100c42dcc8: Status 404 returned error can't find the container with id cb857a29f11bec60cada3302e945c7c0accb94078ef7412aedf62e100c42dcc8 Oct 29 23:08:30.456946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:30.456923 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n"] Oct 29 23:08:30.553984 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:30.553946 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n" event={"ID":"74819147-4b47-42d4-8d03-c12347c309bb","Type":"ContainerStarted","Data":"cb857a29f11bec60cada3302e945c7c0accb94078ef7412aedf62e100c42dcc8"} Oct 29 23:08:31.608211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:31.606683 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n" event={"ID":"74819147-4b47-42d4-8d03-c12347c309bb","Type":"ContainerStarted","Data":"2240aeab2f1a3bb475a06752b52f845eef6f477998537ae701256faa40f481e2"} Oct 29 23:08:31.608211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:31.608027 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n" Oct 29 23:08:32.940032 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:32.939988 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5" podUID="f486535b-541f-4e2a-a5ba-5dfbb9314059" containerName="payload-pre-processing" containerID="cri-o://d0be8ab50080a354810fc94c7f01325f7a476a93405e4b068fff44f074e479d8" gracePeriod=25 Oct 29 23:08:33.250262 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:33.245569 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5" Oct 29 23:08:33.294174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:33.294120 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n" podStartSLOduration=4.294101979 podStartE2EDuration="4.294101979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:31.691518463 +0000 UTC m=+1558.343016994" watchObservedRunningTime="2026-10-29 23:08:33.294101979 +0000 UTC m=+1559.945600511" Oct 29 23:08:33.309714 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:33.309684 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f486535b-541f-4e2a-a5ba-5dfbb9314059-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f486535b-541f-4e2a-a5ba-5dfbb9314059-plugins-config-volume\") pod \"f486535b-541f-4e2a-a5ba-5dfbb9314059\" (UID: \"f486535b-541f-4e2a-a5ba-5dfbb9314059\") " Oct 29 23:08:33.309890 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:33.309747 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f486535b-541f-4e2a-a5ba-5dfbb9314059-kube-api-access-qx7wg\" (UniqueName: \"kubernetes.io/projected/f486535b-541f-4e2a-a5ba-5dfbb9314059-kube-api-access-qx7wg\") pod \"f486535b-541f-4e2a-a5ba-5dfbb9314059\" (UID: \"f486535b-541f-4e2a-a5ba-5dfbb9314059\") " Oct 29 23:08:33.310173 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:33.310145 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f486535b-541f-4e2a-a5ba-5dfbb9314059-plugins-config-volume" pod "f486535b-541f-4e2a-a5ba-5dfbb9314059" (UID: "f486535b-541f-4e2a-a5ba-5dfbb9314059"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:33.310577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:33.310555 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f486535b-541f-4e2a-a5ba-5dfbb9314059-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:33.312024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:33.311994 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f486535b-541f-4e2a-a5ba-5dfbb9314059-kube-api-access-qx7wg" pod "f486535b-541f-4e2a-a5ba-5dfbb9314059" (UID: "f486535b-541f-4e2a-a5ba-5dfbb9314059"). InnerVolumeSpecName "kube-api-access-qx7wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:33.411728 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:33.411695 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx7wg\" (UniqueName: \"kubernetes.io/projected/f486535b-541f-4e2a-a5ba-5dfbb9314059-kube-api-access-qx7wg\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:33.576224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:33.576158 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp"] Oct 29 23:08:33.576469 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:33.576427 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" podUID="3ff64f87-a201-4f46-af24-e1df63c2dab7" containerName="maas-api" containerID="cri-o://14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40" gracePeriod=30 Oct 29 23:08:33.627496 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:33.627462 5057 generic.go:356] "Generic (PLEG): container finished" podID="f486535b-541f-4e2a-a5ba-5dfbb9314059" containerID="d0be8ab50080a354810fc94c7f01325f7a476a93405e4b068fff44f074e479d8" exitCode=0 Oct 29 23:08:33.627700 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:33.627520 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5" Oct 29 23:08:33.627700 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:33.627538 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5" event={"ID":"f486535b-541f-4e2a-a5ba-5dfbb9314059","Type":"ContainerDied","Data":"d0be8ab50080a354810fc94c7f01325f7a476a93405e4b068fff44f074e479d8"} Oct 29 23:08:33.627700 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:33.627593 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5" event={"ID":"f486535b-541f-4e2a-a5ba-5dfbb9314059","Type":"ContainerDied","Data":"c1501007ea0292ce68245c6fde798ba111c74588e60a78331ffdafc1ce9422a7"} Oct 29 23:08:33.627700 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:33.627628 5057 scope.go:122] "RemoveContainer" containerID="d0be8ab50080a354810fc94c7f01325f7a476a93405e4b068fff44f074e479d8" Oct 29 23:08:33.690929 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:33.690905 5057 scope.go:122] "RemoveContainer" containerID="d0be8ab50080a354810fc94c7f01325f7a476a93405e4b068fff44f074e479d8" Oct 29 23:08:33.691315 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:08:33.691292 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0be8ab50080a354810fc94c7f01325f7a476a93405e4b068fff44f074e479d8\": container with ID starting with d0be8ab50080a354810fc94c7f01325f7a476a93405e4b068fff44f074e479d8 not found: ID does not exist" containerID="d0be8ab50080a354810fc94c7f01325f7a476a93405e4b068fff44f074e479d8" Oct 29 23:08:33.691415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:33.691322 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0be8ab50080a354810fc94c7f01325f7a476a93405e4b068fff44f074e479d8"} err="failed to get container status \"d0be8ab50080a354810fc94c7f01325f7a476a93405e4b068fff44f074e479d8\": rpc error: code = NotFound desc = could not find container \"d0be8ab50080a354810fc94c7f01325f7a476a93405e4b068fff44f074e479d8\": container with ID starting with d0be8ab50080a354810fc94c7f01325f7a476a93405e4b068fff44f074e479d8 not found: ID does not exist" Oct 29 23:08:33.695022 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:33.695001 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5"] Oct 29 23:08:33.701564 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:33.701540 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dfedfe-59799c7bd7-7tff5"] Oct 29 23:08:34.301548 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.301198 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" Oct 29 23:08:34.343285 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:08:34.343152 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:08:34.502698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.502658 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff64f87-a201-4f46-af24-e1df63c2dab7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ff64f87-a201-4f46-af24-e1df63c2dab7-metrics-certs\") pod \"3ff64f87-a201-4f46-af24-e1df63c2dab7\" (UID: \"3ff64f87-a201-4f46-af24-e1df63c2dab7\") " Oct 29 23:08:34.502955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.502935 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff64f87-a201-4f46-af24-e1df63c2dab7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ff64f87-a201-4f46-af24-e1df63c2dab7-maas-api-tls\") pod \"3ff64f87-a201-4f46-af24-e1df63c2dab7\" (UID: \"3ff64f87-a201-4f46-af24-e1df63c2dab7\") " Oct 29 23:08:34.503043 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.503027 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff64f87-a201-4f46-af24-e1df63c2dab7-kube-api-access-wndxv\" (UniqueName: \"kubernetes.io/projected/3ff64f87-a201-4f46-af24-e1df63c2dab7-kube-api-access-wndxv\") pod \"3ff64f87-a201-4f46-af24-e1df63c2dab7\" (UID: \"3ff64f87-a201-4f46-af24-e1df63c2dab7\") " Oct 29 23:08:34.503222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.503207 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff64f87-a201-4f46-af24-e1df63c2dab7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff64f87-a201-4f46-af24-e1df63c2dab7-openshift-service-ca\") pod \"3ff64f87-a201-4f46-af24-e1df63c2dab7\" (UID: \"3ff64f87-a201-4f46-af24-e1df63c2dab7\") " Oct 29 23:08:34.503668 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.503641 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff64f87-a201-4f46-af24-e1df63c2dab7-openshift-service-ca" pod "3ff64f87-a201-4f46-af24-e1df63c2dab7" (UID: "3ff64f87-a201-4f46-af24-e1df63c2dab7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:34.504997 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.504977 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff64f87-a201-4f46-af24-e1df63c2dab7-metrics-certs" pod "3ff64f87-a201-4f46-af24-e1df63c2dab7" (UID: "3ff64f87-a201-4f46-af24-e1df63c2dab7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:08:34.505091 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.505049 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff64f87-a201-4f46-af24-e1df63c2dab7-kube-api-access-wndxv" pod "3ff64f87-a201-4f46-af24-e1df63c2dab7" (UID: "3ff64f87-a201-4f46-af24-e1df63c2dab7"). InnerVolumeSpecName "kube-api-access-wndxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:34.505413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.505395 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff64f87-a201-4f46-af24-e1df63c2dab7-maas-api-tls" pod "3ff64f87-a201-4f46-af24-e1df63c2dab7" (UID: "3ff64f87-a201-4f46-af24-e1df63c2dab7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:08:34.605278 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.605245 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ff64f87-a201-4f46-af24-e1df63c2dab7-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:34.605278 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.605273 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wndxv\" (UniqueName: \"kubernetes.io/projected/3ff64f87-a201-4f46-af24-e1df63c2dab7-kube-api-access-wndxv\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:34.605278 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.605284 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff64f87-a201-4f46-af24-e1df63c2dab7-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:34.605514 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.605293 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ff64f87-a201-4f46-af24-e1df63c2dab7-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:34.640824 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.640787 5057 generic.go:356] "Generic (PLEG): container finished" podID="3ff64f87-a201-4f46-af24-e1df63c2dab7" containerID="14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40" exitCode=0 Oct 29 23:08:34.641100 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.640854 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" Oct 29 23:08:34.641100 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.640895 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" event={"ID":"3ff64f87-a201-4f46-af24-e1df63c2dab7","Type":"ContainerDied","Data":"14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40"} Oct 29 23:08:34.641100 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.640942 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp" event={"ID":"3ff64f87-a201-4f46-af24-e1df63c2dab7","Type":"ContainerDied","Data":"b58dcc2c2607b3aa3d83a03e35aa2b211726bf347c1124b033784261ea557ff8"} Oct 29 23:08:34.641100 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.640973 5057 scope.go:122] "RemoveContainer" containerID="14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40" Oct 29 23:08:34.655759 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.655736 5057 scope.go:122] "RemoveContainer" containerID="14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40" Oct 29 23:08:34.656072 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:08:34.656046 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40\": container with ID starting with 14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40 not found: ID does not exist" containerID="14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40" Oct 29 23:08:34.656145 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.656076 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40"} err="failed to get container status \"14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40\": rpc error: code = NotFound desc = could not find container \"14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40\": container with ID starting with 14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40 not found: ID does not exist" Oct 29 23:08:34.700298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.700262 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp"] Oct 29 23:08:34.713402 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:34.713371 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfedfe-5dddb9bf4-wwqnp"] Oct 29 23:08:35.483490 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:35.483456 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff64f87-a201-4f46-af24-e1df63c2dab7" path="/var/lib/kubelet/pods/3ff64f87-a201-4f46-af24-e1df63c2dab7/volumes" Oct 29 23:08:35.483911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:35.483886 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f486535b-541f-4e2a-a5ba-5dfbb9314059" path="/var/lib/kubelet/pods/f486535b-541f-4e2a-a5ba-5dfbb9314059/volumes" Oct 29 23:08:36.245011 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:36.244969 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:08:36.836866 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:36.836830 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw"] Oct 29 23:08:36.837374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:36.837045 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" podUID="4193106b-ad8c-4e1c-bd90-3a244651d979" containerName="istio-proxy" containerID="cri-o://be3117ff97f131af4ed72af5c41127f838cd9b074db501bd2a02a93594085ab0" gracePeriod=30 Oct 29 23:08:38.977623 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:08:38.977586 5057 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff64f87_a201_4f46_af24_e1df63c2dab7.slice/crio-14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:08:39.723264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:39.723228 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc"] Oct 29 23:08:39.723739 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:39.723703 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" podUID="0cb218f5-5d38-4afb-9357-0b2db3da958e" containerName="istio-proxy" containerID="cri-o://c899d69ba8ce6ec5b8a5f76c066cf08750c7eff19b0d0050d6ccdfa8190931e5" gracePeriod=30 Oct 29 23:08:42.025043 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.024992 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d"] Oct 29 23:08:42.025710 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.025649 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" podUID="bec61ff1-da16-48fd-97cc-7a6cc85213e3" containerName="istio-proxy" containerID="cri-o://3f9b2c0b3fe287a05e2ca2c7a4bc65cdac612dd759b006120c91a0ac38202177" gracePeriod=30 Oct 29 23:08:42.111550 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:08:42.104861 5057 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff64f87_a201_4f46_af24_e1df63c2dab7.slice/crio-14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:08:42.269209 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:08:42.268722 5057 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff64f87_a201_4f46_af24_e1df63c2dab7.slice/crio-14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:08:42.538880 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.538852 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 23:08:42.636045 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.636009 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-data\") pod \"4193106b-ad8c-4e1c-bd90-3a244651d979\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " Oct 29 23:08:42.636220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.636068 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4193106b-ad8c-4e1c-bd90-3a244651d979-kube-api-access-8gsqz\" (UniqueName: \"kubernetes.io/projected/4193106b-ad8c-4e1c-bd90-3a244651d979-kube-api-access-8gsqz\") pod \"4193106b-ad8c-4e1c-bd90-3a244651d979\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " Oct 29 23:08:42.636220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.636105 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-workload-socket\") pod \"4193106b-ad8c-4e1c-bd90-3a244651d979\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " Oct 29 23:08:42.636220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.636158 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-token\" (UniqueName: \"kubernetes.io/projected/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-token\") pod \"4193106b-ad8c-4e1c-bd90-3a244651d979\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " Oct 29 23:08:42.636356 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.636338 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-envoy\") pod \"4193106b-ad8c-4e1c-bd90-3a244651d979\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " Oct 29 23:08:42.636543 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.636527 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-workload-certs\") pod \"4193106b-ad8c-4e1c-bd90-3a244651d979\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " Oct 29 23:08:42.636609 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.636563 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-credential-socket\") pod \"4193106b-ad8c-4e1c-bd90-3a244651d979\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " Oct 29 23:08:42.636684 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.636654 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-podinfo\") pod \"4193106b-ad8c-4e1c-bd90-3a244651d979\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " Oct 29 23:08:42.636739 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.636694 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4193106b-ad8c-4e1c-bd90-3a244651d979-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4193106b-ad8c-4e1c-bd90-3a244651d979-istiod-ca-cert\") pod \"4193106b-ad8c-4e1c-bd90-3a244651d979\" (UID: \"4193106b-ad8c-4e1c-bd90-3a244651d979\") " Oct 29 23:08:42.638267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.637865 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4193106b-ad8c-4e1c-bd90-3a244651d979-istiod-ca-cert" pod "4193106b-ad8c-4e1c-bd90-3a244651d979" (UID: "4193106b-ad8c-4e1c-bd90-3a244651d979"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:42.638552 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.638525 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-workload-socket" pod "4193106b-ad8c-4e1c-bd90-3a244651d979" (UID: "4193106b-ad8c-4e1c-bd90-3a244651d979"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:42.638835 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.638812 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-credential-socket" pod "4193106b-ad8c-4e1c-bd90-3a244651d979" (UID: "4193106b-ad8c-4e1c-bd90-3a244651d979"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:42.639013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.638992 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-data" pod "4193106b-ad8c-4e1c-bd90-3a244651d979" (UID: "4193106b-ad8c-4e1c-bd90-3a244651d979"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:42.639344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.639299 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-workload-certs" pod "4193106b-ad8c-4e1c-bd90-3a244651d979" (UID: "4193106b-ad8c-4e1c-bd90-3a244651d979"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:42.640718 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.640685 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4193106b-ad8c-4e1c-bd90-3a244651d979-kube-api-access-8gsqz" pod "4193106b-ad8c-4e1c-bd90-3a244651d979" (UID: "4193106b-ad8c-4e1c-bd90-3a244651d979"). InnerVolumeSpecName "kube-api-access-8gsqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:42.640963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.640897 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-podinfo" pod "4193106b-ad8c-4e1c-bd90-3a244651d979" (UID: "4193106b-ad8c-4e1c-bd90-3a244651d979"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:08:42.641289 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.641248 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-token" pod "4193106b-ad8c-4e1c-bd90-3a244651d979" (UID: "4193106b-ad8c-4e1c-bd90-3a244651d979"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:42.643435 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.643406 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-envoy" pod "4193106b-ad8c-4e1c-bd90-3a244651d979" (UID: "4193106b-ad8c-4e1c-bd90-3a244651d979"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:42.738933 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.738692 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4193106b-ad8c-4e1c-bd90-3a244651d979-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:42.738933 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.738730 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:42.738933 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.738744 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gsqz\" (UniqueName: \"kubernetes.io/projected/4193106b-ad8c-4e1c-bd90-3a244651d979-kube-api-access-8gsqz\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:42.738933 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.738759 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:42.738933 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.738774 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:42.738933 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.738788 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:42.738933 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.738800 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:42.738933 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.738814 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4193106b-ad8c-4e1c-bd90-3a244651d979-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:42.738933 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.738827 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4193106b-ad8c-4e1c-bd90-3a244651d979-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:42.744946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.744913 5057 generic.go:356] "Generic (PLEG): container finished" podID="4193106b-ad8c-4e1c-bd90-3a244651d979" containerID="be3117ff97f131af4ed72af5c41127f838cd9b074db501bd2a02a93594085ab0" exitCode=0 Oct 29 23:08:42.745088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.744971 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" Oct 29 23:08:42.745088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.744999 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" event={"ID":"4193106b-ad8c-4e1c-bd90-3a244651d979","Type":"ContainerDied","Data":"be3117ff97f131af4ed72af5c41127f838cd9b074db501bd2a02a93594085ab0"} Oct 29 23:08:42.745170 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.745086 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw" event={"ID":"4193106b-ad8c-4e1c-bd90-3a244651d979","Type":"ContainerDied","Data":"ddf74988d336d72c634b1818d6dd077c86e71b8def59d042272fdeb105ca6de6"} Oct 29 23:08:42.745170 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.745110 5057 scope.go:122] "RemoveContainer" containerID="be3117ff97f131af4ed72af5c41127f838cd9b074db501bd2a02a93594085ab0" Oct 29 23:08:42.756481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.756465 5057 scope.go:122] "RemoveContainer" containerID="be3117ff97f131af4ed72af5c41127f838cd9b074db501bd2a02a93594085ab0" Oct 29 23:08:42.756730 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:08:42.756714 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be3117ff97f131af4ed72af5c41127f838cd9b074db501bd2a02a93594085ab0\": container with ID starting with be3117ff97f131af4ed72af5c41127f838cd9b074db501bd2a02a93594085ab0 not found: ID does not exist" containerID="be3117ff97f131af4ed72af5c41127f838cd9b074db501bd2a02a93594085ab0" Oct 29 23:08:42.756777 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.756737 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3117ff97f131af4ed72af5c41127f838cd9b074db501bd2a02a93594085ab0"} err="failed to get container status \"be3117ff97f131af4ed72af5c41127f838cd9b074db501bd2a02a93594085ab0\": rpc error: code = NotFound desc = could not find container \"be3117ff97f131af4ed72af5c41127f838cd9b074db501bd2a02a93594085ab0\": container with ID starting with be3117ff97f131af4ed72af5c41127f838cd9b074db501bd2a02a93594085ab0 not found: ID does not exist" Oct 29 23:08:42.804793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.804759 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw"] Oct 29 23:08:42.808745 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:42.808721 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bbxhw"] Oct 29 23:08:43.482939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:43.482905 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4193106b-ad8c-4e1c-bd90-3a244651d979" path="/var/lib/kubelet/pods/4193106b-ad8c-4e1c-bd90-3a244651d979/volumes" Oct 29 23:08:43.652263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:43.650399 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n" Oct 29 23:08:45.064487 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.064461 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:08:45.175663 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.175623 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-envoy\") pod \"0cb218f5-5d38-4afb-9357-0b2db3da958e\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " Oct 29 23:08:45.175862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.175674 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-podinfo\") pod \"0cb218f5-5d38-4afb-9357-0b2db3da958e\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " Oct 29 23:08:45.175862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.175758 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-workload-socket\") pod \"0cb218f5-5d38-4afb-9357-0b2db3da958e\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " Oct 29 23:08:45.175981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.175957 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb218f5-5d38-4afb-9357-0b2db3da958e-kube-api-access-f7p85\" (UniqueName: \"kubernetes.io/projected/0cb218f5-5d38-4afb-9357-0b2db3da958e-kube-api-access-f7p85\") pod \"0cb218f5-5d38-4afb-9357-0b2db3da958e\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " Oct 29 23:08:45.176049 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.176019 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-data\") pod \"0cb218f5-5d38-4afb-9357-0b2db3da958e\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " Oct 29 23:08:45.176132 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.176112 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-workload-certs\") pod \"0cb218f5-5d38-4afb-9357-0b2db3da958e\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " Oct 29 23:08:45.176252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.176163 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-token\" (UniqueName: \"kubernetes.io/projected/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-token\") pod \"0cb218f5-5d38-4afb-9357-0b2db3da958e\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " Oct 29 23:08:45.176252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.176243 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-credential-socket\") pod \"0cb218f5-5d38-4afb-9357-0b2db3da958e\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " Oct 29 23:08:45.176388 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.176369 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb218f5-5d38-4afb-9357-0b2db3da958e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cb218f5-5d38-4afb-9357-0b2db3da958e-istiod-ca-cert\") pod \"0cb218f5-5d38-4afb-9357-0b2db3da958e\" (UID: \"0cb218f5-5d38-4afb-9357-0b2db3da958e\") " Oct 29 23:08:45.176463 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.176385 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-data" pod "0cb218f5-5d38-4afb-9357-0b2db3da958e" (UID: "0cb218f5-5d38-4afb-9357-0b2db3da958e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:45.176463 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.176399 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-workload-socket" pod "0cb218f5-5d38-4afb-9357-0b2db3da958e" (UID: "0cb218f5-5d38-4afb-9357-0b2db3da958e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:45.176615 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.176590 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-credential-socket" pod "0cb218f5-5d38-4afb-9357-0b2db3da958e" (UID: "0cb218f5-5d38-4afb-9357-0b2db3da958e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:45.176615 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.176606 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-workload-certs" pod "0cb218f5-5d38-4afb-9357-0b2db3da958e" (UID: "0cb218f5-5d38-4afb-9357-0b2db3da958e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:45.176787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.176768 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb218f5-5d38-4afb-9357-0b2db3da958e-istiod-ca-cert" pod "0cb218f5-5d38-4afb-9357-0b2db3da958e" (UID: "0cb218f5-5d38-4afb-9357-0b2db3da958e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:45.177028 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.177011 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:45.177028 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.177029 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:45.177250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.177038 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:45.177250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.177046 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cb218f5-5d38-4afb-9357-0b2db3da958e-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:45.177250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.177058 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:45.178865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.178689 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-envoy" pod "0cb218f5-5d38-4afb-9357-0b2db3da958e" (UID: "0cb218f5-5d38-4afb-9357-0b2db3da958e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:45.179159 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.179132 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-podinfo" pod "0cb218f5-5d38-4afb-9357-0b2db3da958e" (UID: "0cb218f5-5d38-4afb-9357-0b2db3da958e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:08:45.179505 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.179467 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-token" pod "0cb218f5-5d38-4afb-9357-0b2db3da958e" (UID: "0cb218f5-5d38-4afb-9357-0b2db3da958e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:45.179593 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.179501 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb218f5-5d38-4afb-9357-0b2db3da958e-kube-api-access-f7p85" pod "0cb218f5-5d38-4afb-9357-0b2db3da958e" (UID: "0cb218f5-5d38-4afb-9357-0b2db3da958e"). InnerVolumeSpecName "kube-api-access-f7p85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:45.279260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.278355 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7p85\" (UniqueName: \"kubernetes.io/projected/0cb218f5-5d38-4afb-9357-0b2db3da958e-kube-api-access-f7p85\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:45.279260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.278390 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:45.279260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.278406 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:45.279260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.278421 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cb218f5-5d38-4afb-9357-0b2db3da958e-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:45.758163 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.758130 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:08:45.758347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.758225 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" podUID="bec61ff1-da16-48fd-97cc-7a6cc85213e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:08:45.777297 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.777262 5057 generic.go:356] "Generic (PLEG): container finished" podID="0cb218f5-5d38-4afb-9357-0b2db3da958e" containerID="c899d69ba8ce6ec5b8a5f76c066cf08750c7eff19b0d0050d6ccdfa8190931e5" exitCode=0 Oct 29 23:08:45.777473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.777318 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" Oct 29 23:08:45.777473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.777345 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" event={"ID":"0cb218f5-5d38-4afb-9357-0b2db3da958e","Type":"ContainerDied","Data":"c899d69ba8ce6ec5b8a5f76c066cf08750c7eff19b0d0050d6ccdfa8190931e5"} Oct 29 23:08:45.777473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.777419 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc" event={"ID":"0cb218f5-5d38-4afb-9357-0b2db3da958e","Type":"ContainerDied","Data":"b58f118d16b6f115e687405a00050c1413b0c4e699b8aeb0117f6065442d1e14"} Oct 29 23:08:45.777473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.777449 5057 scope.go:122] "RemoveContainer" containerID="c899d69ba8ce6ec5b8a5f76c066cf08750c7eff19b0d0050d6ccdfa8190931e5" Oct 29 23:08:45.789879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.789850 5057 scope.go:122] "RemoveContainer" containerID="73a3f54198ffc6a3e5f5d9850cb7f9890874877f74fee6af44d39047b93ae598" Oct 29 23:08:45.801099 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.801081 5057 scope.go:122] "RemoveContainer" containerID="c899d69ba8ce6ec5b8a5f76c066cf08750c7eff19b0d0050d6ccdfa8190931e5" Oct 29 23:08:45.801396 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:08:45.801377 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c899d69ba8ce6ec5b8a5f76c066cf08750c7eff19b0d0050d6ccdfa8190931e5\": container with ID starting with c899d69ba8ce6ec5b8a5f76c066cf08750c7eff19b0d0050d6ccdfa8190931e5 not found: ID does not exist" containerID="c899d69ba8ce6ec5b8a5f76c066cf08750c7eff19b0d0050d6ccdfa8190931e5" Oct 29 23:08:45.801468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.801404 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c899d69ba8ce6ec5b8a5f76c066cf08750c7eff19b0d0050d6ccdfa8190931e5"} err="failed to get container status \"c899d69ba8ce6ec5b8a5f76c066cf08750c7eff19b0d0050d6ccdfa8190931e5\": rpc error: code = NotFound desc = could not find container \"c899d69ba8ce6ec5b8a5f76c066cf08750c7eff19b0d0050d6ccdfa8190931e5\": container with ID starting with c899d69ba8ce6ec5b8a5f76c066cf08750c7eff19b0d0050d6ccdfa8190931e5 not found: ID does not exist" Oct 29 23:08:45.801468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.801422 5057 scope.go:122] "RemoveContainer" containerID="73a3f54198ffc6a3e5f5d9850cb7f9890874877f74fee6af44d39047b93ae598" Oct 29 23:08:45.801693 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:08:45.801668 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a3f54198ffc6a3e5f5d9850cb7f9890874877f74fee6af44d39047b93ae598\": container with ID starting with 73a3f54198ffc6a3e5f5d9850cb7f9890874877f74fee6af44d39047b93ae598 not found: ID does not exist" containerID="73a3f54198ffc6a3e5f5d9850cb7f9890874877f74fee6af44d39047b93ae598" Oct 29 23:08:45.801739 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.801703 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a3f54198ffc6a3e5f5d9850cb7f9890874877f74fee6af44d39047b93ae598"} err="failed to get container status \"73a3f54198ffc6a3e5f5d9850cb7f9890874877f74fee6af44d39047b93ae598\": rpc error: code = NotFound desc = could not find container \"73a3f54198ffc6a3e5f5d9850cb7f9890874877f74fee6af44d39047b93ae598\": container with ID starting with 73a3f54198ffc6a3e5f5d9850cb7f9890874877f74fee6af44d39047b93ae598 not found: ID does not exist" Oct 29 23:08:45.854796 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.854766 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc"] Oct 29 23:08:45.874407 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:45.874370 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-ce5978-openshift-default-7974489469-s6crc"] Oct 29 23:08:47.314263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.314237 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:08:47.396319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.396280 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec61ff1-da16-48fd-97cc-7a6cc85213e3-kube-api-access-vbpqn\" (UniqueName: \"kubernetes.io/projected/bec61ff1-da16-48fd-97cc-7a6cc85213e3-kube-api-access-vbpqn\") pod \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " Oct 29 23:08:47.396487 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.396330 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-envoy\") pod \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " Oct 29 23:08:47.396487 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.396454 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istiod-ca-cert\") pod \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " Oct 29 23:08:47.396652 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.396629 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-workload-socket\") pod \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " Oct 29 23:08:47.396751 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.396721 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-data\") pod \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " Oct 29 23:08:47.396818 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.396776 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-workload-certs\") pod \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " Oct 29 23:08:47.396875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.396818 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-podinfo\") pod \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " Oct 29 23:08:47.396875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.396837 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istiod-ca-cert" pod "bec61ff1-da16-48fd-97cc-7a6cc85213e3" (UID: "bec61ff1-da16-48fd-97cc-7a6cc85213e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:47.396971 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.396904 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-credential-socket\") pod \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " Oct 29 23:08:47.397025 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.396990 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-data" pod "bec61ff1-da16-48fd-97cc-7a6cc85213e3" (UID: "bec61ff1-da16-48fd-97cc-7a6cc85213e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:47.397084 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.397001 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-token\" (UniqueName: \"kubernetes.io/projected/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-token\") pod \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\" (UID: \"bec61ff1-da16-48fd-97cc-7a6cc85213e3\") " Oct 29 23:08:47.397142 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.397087 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-workload-certs" pod "bec61ff1-da16-48fd-97cc-7a6cc85213e3" (UID: "bec61ff1-da16-48fd-97cc-7a6cc85213e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:47.397298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.397272 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-credential-socket" pod "bec61ff1-da16-48fd-97cc-7a6cc85213e3" (UID: "bec61ff1-da16-48fd-97cc-7a6cc85213e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:47.397525 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.397486 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-workload-socket" pod "bec61ff1-da16-48fd-97cc-7a6cc85213e3" (UID: "bec61ff1-da16-48fd-97cc-7a6cc85213e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:47.397860 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.397834 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:47.397981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.397864 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:47.397981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.397880 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:47.397981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.397893 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:47.397981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.397907 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:47.399492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.399469 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-podinfo" pod "bec61ff1-da16-48fd-97cc-7a6cc85213e3" (UID: "bec61ff1-da16-48fd-97cc-7a6cc85213e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:08:47.400594 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.400573 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-envoy" pod "bec61ff1-da16-48fd-97cc-7a6cc85213e3" (UID: "bec61ff1-da16-48fd-97cc-7a6cc85213e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:47.400700 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.400683 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-token" pod "bec61ff1-da16-48fd-97cc-7a6cc85213e3" (UID: "bec61ff1-da16-48fd-97cc-7a6cc85213e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:47.401675 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.401645 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec61ff1-da16-48fd-97cc-7a6cc85213e3-kube-api-access-vbpqn" pod "bec61ff1-da16-48fd-97cc-7a6cc85213e3" (UID: "bec61ff1-da16-48fd-97cc-7a6cc85213e3"). InnerVolumeSpecName "kube-api-access-vbpqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:47.483663 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.483629 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cb218f5-5d38-4afb-9357-0b2db3da958e" path="/var/lib/kubelet/pods/0cb218f5-5d38-4afb-9357-0b2db3da958e/volumes" Oct 29 23:08:47.499307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.499278 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:47.499307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.499309 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:47.499479 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.499392 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbpqn\" (UniqueName: \"kubernetes.io/projected/bec61ff1-da16-48fd-97cc-7a6cc85213e3-kube-api-access-vbpqn\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:47.499479 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.499411 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec61ff1-da16-48fd-97cc-7a6cc85213e3-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:08:47.795169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.795129 5057 generic.go:356] "Generic (PLEG): container finished" podID="bec61ff1-da16-48fd-97cc-7a6cc85213e3" containerID="3f9b2c0b3fe287a05e2ca2c7a4bc65cdac612dd759b006120c91a0ac38202177" exitCode=0 Oct 29 23:08:47.795393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.795212 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" Oct 29 23:08:47.795393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.795220 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" event={"ID":"bec61ff1-da16-48fd-97cc-7a6cc85213e3","Type":"ContainerDied","Data":"3f9b2c0b3fe287a05e2ca2c7a4bc65cdac612dd759b006120c91a0ac38202177"} Oct 29 23:08:47.795393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.795277 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d" event={"ID":"bec61ff1-da16-48fd-97cc-7a6cc85213e3","Type":"ContainerDied","Data":"de5994c7299bc7382d29d08c681ec982a5d113ba6b7df72fb17bdf80c86eaf18"} Oct 29 23:08:47.795393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.795301 5057 scope.go:122] "RemoveContainer" containerID="3f9b2c0b3fe287a05e2ca2c7a4bc65cdac612dd759b006120c91a0ac38202177" Oct 29 23:08:47.809918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.809876 5057 scope.go:122] "RemoveContainer" containerID="d97a49bf83f1840cf074f16f1e08c6855c6c11f810cc19211ea2be9eb73e3334" Oct 29 23:08:47.820020 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.819999 5057 scope.go:122] "RemoveContainer" containerID="3f9b2c0b3fe287a05e2ca2c7a4bc65cdac612dd759b006120c91a0ac38202177" Oct 29 23:08:47.820353 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:08:47.820332 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9b2c0b3fe287a05e2ca2c7a4bc65cdac612dd759b006120c91a0ac38202177\": container with ID starting with 3f9b2c0b3fe287a05e2ca2c7a4bc65cdac612dd759b006120c91a0ac38202177 not found: ID does not exist" containerID="3f9b2c0b3fe287a05e2ca2c7a4bc65cdac612dd759b006120c91a0ac38202177" Oct 29 23:08:47.820402 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.820364 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9b2c0b3fe287a05e2ca2c7a4bc65cdac612dd759b006120c91a0ac38202177"} err="failed to get container status \"3f9b2c0b3fe287a05e2ca2c7a4bc65cdac612dd759b006120c91a0ac38202177\": rpc error: code = NotFound desc = could not find container \"3f9b2c0b3fe287a05e2ca2c7a4bc65cdac612dd759b006120c91a0ac38202177\": container with ID starting with 3f9b2c0b3fe287a05e2ca2c7a4bc65cdac612dd759b006120c91a0ac38202177 not found: ID does not exist" Oct 29 23:08:47.820402 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.820381 5057 scope.go:122] "RemoveContainer" containerID="d97a49bf83f1840cf074f16f1e08c6855c6c11f810cc19211ea2be9eb73e3334" Oct 29 23:08:47.820639 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:08:47.820621 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97a49bf83f1840cf074f16f1e08c6855c6c11f810cc19211ea2be9eb73e3334\": container with ID starting with d97a49bf83f1840cf074f16f1e08c6855c6c11f810cc19211ea2be9eb73e3334 not found: ID does not exist" containerID="d97a49bf83f1840cf074f16f1e08c6855c6c11f810cc19211ea2be9eb73e3334" Oct 29 23:08:47.820684 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.820647 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97a49bf83f1840cf074f16f1e08c6855c6c11f810cc19211ea2be9eb73e3334"} err="failed to get container status \"d97a49bf83f1840cf074f16f1e08c6855c6c11f810cc19211ea2be9eb73e3334\": rpc error: code = NotFound desc = could not find container \"d97a49bf83f1840cf074f16f1e08c6855c6c11f810cc19211ea2be9eb73e3334\": container with ID starting with d97a49bf83f1840cf074f16f1e08c6855c6c11f810cc19211ea2be9eb73e3334 not found: ID does not exist" Oct 29 23:08:47.857831 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.857797 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d"] Oct 29 23:08:47.869935 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:47.869908 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-dfedfe-openshift-default-6547c7754f-gwx2d"] Oct 29 23:08:48.537541 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:08:48.537506 5057 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff64f87_a201_4f46_af24_e1df63c2dab7.slice/crio-14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:08:48.994800 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:08:48.994762 5057 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff64f87_a201_4f46_af24_e1df63c2dab7.slice/crio-14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:08:49.482481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:08:49.482450 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bec61ff1-da16-48fd-97cc-7a6cc85213e3" path="/var/lib/kubelet/pods/bec61ff1-da16-48fd-97cc-7a6cc85213e3/volumes" Oct 29 23:08:59.190815 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:08:59.190783 5057 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff64f87_a201_4f46_af24_e1df63c2dab7.slice/crio-14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:09:02.714180 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.714110 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl"] Oct 29 23:09:02.714646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.714461 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" podUID="379d09bf-6153-4be5-be07-e5ba38818871" containerName="istio-proxy" containerID="cri-o://e798c22aa47acbd74128534d604ac1e9702a34370c4c9e5fb74ac5eca5667442" gracePeriod=30 Oct 29 23:09:02.955210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.955070 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl"] Oct 29 23:09:02.956232 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.956173 5057 cpu_manager.go:414] "containerMap: removing container" podUID="bec61ff1-da16-48fd-97cc-7a6cc85213e3" containerName="istio-proxy" Oct 29 23:09:02.956232 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.956211 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec61ff1-da16-48fd-97cc-7a6cc85213e3" containerName="istio-proxy" Oct 29 23:09:02.956232 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.956231 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4193106b-ad8c-4e1c-bd90-3a244651d979" containerName="istio-proxy" Oct 29 23:09:02.956464 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.956239 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4193106b-ad8c-4e1c-bd90-3a244651d979" containerName="istio-proxy" Oct 29 23:09:02.956464 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.956262 5057 cpu_manager.go:414] "containerMap: removing container" podUID="0cb218f5-5d38-4afb-9357-0b2db3da958e" containerName="istio-proxy" Oct 29 23:09:02.956464 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.956270 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb218f5-5d38-4afb-9357-0b2db3da958e" containerName="istio-proxy" Oct 29 23:09:02.956464 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.956297 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f486535b-541f-4e2a-a5ba-5dfbb9314059" containerName="payload-pre-processing" Oct 29 23:09:02.956464 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.956306 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f486535b-541f-4e2a-a5ba-5dfbb9314059" containerName="payload-pre-processing" Oct 29 23:09:02.956464 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.956350 5057 cpu_manager.go:414] "containerMap: removing container" podUID="3ff64f87-a201-4f46-af24-e1df63c2dab7" containerName="maas-api" Oct 29 23:09:02.956464 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.956357 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff64f87-a201-4f46-af24-e1df63c2dab7" containerName="maas-api" Oct 29 23:09:02.956868 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.956582 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f486535b-541f-4e2a-a5ba-5dfbb9314059" containerName="payload-pre-processing" Oct 29 23:09:02.956868 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.956602 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec61ff1-da16-48fd-97cc-7a6cc85213e3" containerName="istio-proxy" Oct 29 23:09:02.956868 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.956615 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff64f87-a201-4f46-af24-e1df63c2dab7" containerName="maas-api" Oct 29 23:09:02.956868 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.956627 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb218f5-5d38-4afb-9357-0b2db3da958e" containerName="istio-proxy" Oct 29 23:09:02.956868 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.956642 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4193106b-ad8c-4e1c-bd90-3a244651d979" containerName="istio-proxy" Oct 29 23:09:02.961263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.961231 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:02.983291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.982566 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-85c702-openshift-default-dockercfg-vhccb\"" Oct 29 23:09:02.986643 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:02.986622 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl"] Oct 29 23:09:03.009612 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.009575 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-token\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.009853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.009648 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b4ea44-8863-40e5-baee-a5e1e0c94285-istiod-ca-cert\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.009853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.009746 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-envoy\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.010008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.009865 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7lr9\" (UniqueName: \"kubernetes.io/projected/72b4ea44-8863-40e5-baee-a5e1e0c94285-kube-api-access-x7lr9\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.010008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.009896 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-workload-certs\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.010008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.009962 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-credential-socket\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.010008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.009993 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-podinfo\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.010235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.010066 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-data\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.010235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.010095 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-workload-socket\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.111650 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.111607 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-token\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.111844 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.111665 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b4ea44-8863-40e5-baee-a5e1e0c94285-istiod-ca-cert\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.111844 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.111696 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-envoy\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.112046 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.112021 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7lr9\" (UniqueName: \"kubernetes.io/projected/72b4ea44-8863-40e5-baee-a5e1e0c94285-kube-api-access-x7lr9\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.112108 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.112066 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-workload-certs\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.112164 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.112147 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-credential-socket\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.112259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.112181 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-podinfo\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.112336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.112284 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-data\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.112336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.112318 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-workload-socket\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.112552 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.112503 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-workload-certs\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.112629 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.112582 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b4ea44-8863-40e5-baee-a5e1e0c94285-istiod-ca-cert\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.113027 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.112712 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-workload-socket\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.113027 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.112893 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-credential-socket\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.113550 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.113527 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-data\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.114902 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.114861 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-podinfo\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.115003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.114974 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-envoy\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.143223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.141343 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-token\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.143223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.143154 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7lr9\" (UniqueName: \"kubernetes.io/projected/72b4ea44-8863-40e5-baee-a5e1e0c94285-kube-api-access-x7lr9\") pod \"e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.286884 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.286513 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:03.474119 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:09:03.473211 5057 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff64f87_a201_4f46_af24_e1df63c2dab7.slice/crio-14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:09:03.825024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.814169 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl"] Oct 29 23:09:03.865915 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:09:03.865873 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b4ea44_8863_40e5_baee_a5e1e0c94285.slice/crio-32f50860274d7ba321c8e4074c26d95460a28d82b68d74d9769aa4f96b82542d WatchSource:0}: Error finding container 32f50860274d7ba321c8e4074c26d95460a28d82b68d74d9769aa4f96b82542d: Status 404 returned error can't find the container with id 32f50860274d7ba321c8e4074c26d95460a28d82b68d74d9769aa4f96b82542d Oct 29 23:09:03.870664 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.870623 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:09:03.871281 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.871251 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:09:03.871451 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:03.871432 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:09:03.947346 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:09:03.947302 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:09:04.015167 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:04.014810 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" event={"ID":"72b4ea44-8863-40e5-baee-a5e1e0c94285","Type":"ContainerStarted","Data":"9f0fd9f5b13f83680835610c4c9f4b45f72d05c529fff22ee16a2fdc83456df4"} Oct 29 23:09:04.015167 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:04.014902 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" event={"ID":"72b4ea44-8863-40e5-baee-a5e1e0c94285","Type":"ContainerStarted","Data":"32f50860274d7ba321c8e4074c26d95460a28d82b68d74d9769aa4f96b82542d"} Oct 29 23:09:04.073997 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:04.073926 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" podStartSLOduration=2.073905292 podStartE2EDuration="2.073905292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:04.068223741 +0000 UTC m=+1590.719722285" watchObservedRunningTime="2026-10-29 23:09:04.073905292 +0000 UTC m=+1590.725403824" Oct 29 23:09:04.303215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:04.294067 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 29 23:09:04.303215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:04.294173 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" podUID="72b4ea44-8863-40e5-baee-a5e1e0c94285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 29 23:09:04.303215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:04.294573 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:04.699689 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:04.699655 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59"] Oct 29 23:09:04.700229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:04.700158 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" podUID="fdd750a6-8487-4d03-9e52-4c6b6ef9133b" containerName="istio-proxy" containerID="cri-o://6cd18c7f34df3865986b41baca418b6ff2c4ea914420ff8a0112a95789549aa1" gracePeriod=30 Oct 29 23:09:05.294363 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:05.294326 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:06.054829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.054788 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:06.055965 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.055940 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:09:06.796814 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.796775 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf"] Oct 29 23:09:06.798023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.797995 5057 cpu_manager.go:414] "containerMap: removing container" podUID="bec61ff1-da16-48fd-97cc-7a6cc85213e3" containerName="istio-proxy" Oct 29 23:09:06.798023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.798021 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec61ff1-da16-48fd-97cc-7a6cc85213e3" containerName="istio-proxy" Oct 29 23:09:06.798263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.798055 5057 cpu_manager.go:414] "containerMap: removing container" podUID="0cb218f5-5d38-4afb-9357-0b2db3da958e" containerName="istio-proxy" Oct 29 23:09:06.798263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.798064 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb218f5-5d38-4afb-9357-0b2db3da958e" containerName="istio-proxy" Oct 29 23:09:06.798384 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.798309 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec61ff1-da16-48fd-97cc-7a6cc85213e3" containerName="istio-proxy" Oct 29 23:09:06.798384 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.798334 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb218f5-5d38-4afb-9357-0b2db3da958e" containerName="istio-proxy" Oct 29 23:09:06.803237 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.803214 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.833940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.832957 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-credential-socket\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.833940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.833073 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-workload-certs\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.833940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.833115 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-workload-socket\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.833940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.833138 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-envoy\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.833940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.833234 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-token\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.833940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.833275 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-data\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.833940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.833324 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-podinfo\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.833940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.833374 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istiod-ca-cert\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.833940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.833414 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dttcp\" (UniqueName: \"kubernetes.io/projected/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-kube-api-access-dttcp\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.850630 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.839934 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf"] Oct 29 23:09:06.850630 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.842655 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-f2c8d3-openshift-default-dockercfg-xpgjj\"" Oct 29 23:09:06.949221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.948124 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dttcp\" (UniqueName: \"kubernetes.io/projected/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-kube-api-access-dttcp\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.949221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.948256 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-credential-socket\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.949221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.948375 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-workload-certs\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.949221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.948421 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-workload-socket\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.949221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.948444 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-envoy\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.949221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.948537 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-token\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.949221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.948585 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-data\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.949221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.948640 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-podinfo\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.949221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.948695 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istiod-ca-cert\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.949879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.949521 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istiod-ca-cert\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.960225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.950102 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-credential-socket\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.960225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.950347 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-workload-certs\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:06.960225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.950557 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-workload-socket\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:07.012892 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:06.983523 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-data\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:07.098259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:07.053752 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dttcp\" (UniqueName: \"kubernetes.io/projected/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-kube-api-access-dttcp\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:07.098259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:07.057083 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-envoy\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:07.135956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:07.128773 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-token\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:07.206957 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:07.197768 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-podinfo\") pod \"e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:07.219335 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:07.218513 5057 patch_prober.go:29] interesting pod/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:09:07.219335 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:07.218584 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" podUID="fdd750a6-8487-4d03-9e52-4c6b6ef9133b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:09:07.445130 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:07.445046 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:07.805936 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:07.805906 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf"] Oct 29 23:09:07.815368 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:09:07.815335 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82365a6f_eb57_45d8_bcd4_ff4e8daddf14.slice/crio-296b52db72d8b014bab1c76b7d08a6543ee307e34f5293e60dbd8368b41f1dfc WatchSource:0}: Error finding container 296b52db72d8b014bab1c76b7d08a6543ee307e34f5293e60dbd8368b41f1dfc: Status 404 returned error can't find the container with id 296b52db72d8b014bab1c76b7d08a6543ee307e34f5293e60dbd8368b41f1dfc Oct 29 23:09:07.818309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:07.818272 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:09:07.818428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:07.818358 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:09:07.818428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:07.818407 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:09:08.084528 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.084436 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 29 23:09:08.084528 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.084505 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" podUID="379d09bf-6153-4be5-be07-e5ba38818871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 29 23:09:08.180280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.180235 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" event={"ID":"82365a6f-eb57-45d8-bcd4-ff4e8daddf14","Type":"ContainerStarted","Data":"760b92627049a7308e812ccd98760a670aeb09ff04746396f39b74ca578a090f"} Oct 29 23:09:08.180481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.180309 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" event={"ID":"82365a6f-eb57-45d8-bcd4-ff4e8daddf14","Type":"ContainerStarted","Data":"296b52db72d8b014bab1c76b7d08a6543ee307e34f5293e60dbd8368b41f1dfc"} Oct 29 23:09:08.186209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.184230 5057 generic.go:356] "Generic (PLEG): container finished" podID="379d09bf-6153-4be5-be07-e5ba38818871" containerID="e798c22aa47acbd74128534d604ac1e9702a34370c4c9e5fb74ac5eca5667442" exitCode=0 Oct 29 23:09:08.186209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.185319 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" event={"ID":"379d09bf-6153-4be5-be07-e5ba38818871","Type":"ContainerDied","Data":"e798c22aa47acbd74128534d604ac1e9702a34370c4c9e5fb74ac5eca5667442"} Oct 29 23:09:08.246400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.246153 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" podStartSLOduration=2.246132349 podStartE2EDuration="2.246132349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:08.242784867 +0000 UTC m=+1594.894283399" watchObservedRunningTime="2026-10-29 23:09:08.246132349 +0000 UTC m=+1594.897630879" Oct 29 23:09:08.446048 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.445957 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:08.460276 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.453429 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:08.810549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.809097 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 23:09:08.922915 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.922826 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/379d09bf-6153-4be5-be07-e5ba38818871-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379d09bf-6153-4be5-be07-e5ba38818871-istiod-ca-cert\") pod \"379d09bf-6153-4be5-be07-e5ba38818871\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " Oct 29 23:09:08.923086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.922920 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/379d09bf-6153-4be5-be07-e5ba38818871-istio-token\" (UniqueName: \"kubernetes.io/projected/379d09bf-6153-4be5-be07-e5ba38818871-istio-token\") pod \"379d09bf-6153-4be5-be07-e5ba38818871\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " Oct 29 23:09:08.923330 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.923287 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/379d09bf-6153-4be5-be07-e5ba38818871-istiod-ca-cert" pod "379d09bf-6153-4be5-be07-e5ba38818871" (UID: "379d09bf-6153-4be5-be07-e5ba38818871"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:09:08.923688 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.923021 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/379d09bf-6153-4be5-be07-e5ba38818871-kube-api-access-rd6p9\" (UniqueName: \"kubernetes.io/projected/379d09bf-6153-4be5-be07-e5ba38818871-kube-api-access-rd6p9\") pod \"379d09bf-6153-4be5-be07-e5ba38818871\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " Oct 29 23:09:08.924138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.923830 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/379d09bf-6153-4be5-be07-e5ba38818871-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379d09bf-6153-4be5-be07-e5ba38818871-istio-podinfo\") pod \"379d09bf-6153-4be5-be07-e5ba38818871\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " Oct 29 23:09:08.924138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.923903 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-credential-socket\") pod \"379d09bf-6153-4be5-be07-e5ba38818871\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " Oct 29 23:09:08.924138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.923955 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-istio-envoy\") pod \"379d09bf-6153-4be5-be07-e5ba38818871\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " Oct 29 23:09:08.924138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.924005 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-workload-certs\") pod \"379d09bf-6153-4be5-be07-e5ba38818871\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " Oct 29 23:09:08.924138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.924059 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-workload-socket\") pod \"379d09bf-6153-4be5-be07-e5ba38818871\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " Oct 29 23:09:08.924138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.924107 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-istio-data\") pod \"379d09bf-6153-4be5-be07-e5ba38818871\" (UID: \"379d09bf-6153-4be5-be07-e5ba38818871\") " Oct 29 23:09:08.924829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.924801 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-workload-socket" pod "379d09bf-6153-4be5-be07-e5ba38818871" (UID: "379d09bf-6153-4be5-be07-e5ba38818871"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:08.924961 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.924883 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-workload-certs" pod "379d09bf-6153-4be5-be07-e5ba38818871" (UID: "379d09bf-6153-4be5-be07-e5ba38818871"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:08.924961 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.924949 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-credential-socket" pod "379d09bf-6153-4be5-be07-e5ba38818871" (UID: "379d09bf-6153-4be5-be07-e5ba38818871"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:08.925299 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.925278 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-istio-data" pod "379d09bf-6153-4be5-be07-e5ba38818871" (UID: "379d09bf-6153-4be5-be07-e5ba38818871"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:08.925711 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.925682 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379d09bf-6153-4be5-be07-e5ba38818871-istio-token" pod "379d09bf-6153-4be5-be07-e5ba38818871" (UID: "379d09bf-6153-4be5-be07-e5ba38818871"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:09:08.925979 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.925953 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379d09bf-6153-4be5-be07-e5ba38818871-kube-api-access-rd6p9" pod "379d09bf-6153-4be5-be07-e5ba38818871" (UID: "379d09bf-6153-4be5-be07-e5ba38818871"). InnerVolumeSpecName "kube-api-access-rd6p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:09:08.926095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.925986 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:08.926095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.926004 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:08.926095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.926016 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:08.926095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.926026 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:08.926095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.926034 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379d09bf-6153-4be5-be07-e5ba38818871-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:08.926095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.926042 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/379d09bf-6153-4be5-be07-e5ba38818871-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:08.926797 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.926772 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-istio-envoy" pod "379d09bf-6153-4be5-be07-e5ba38818871" (UID: "379d09bf-6153-4be5-be07-e5ba38818871"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:08.928729 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:08.926949 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/379d09bf-6153-4be5-be07-e5ba38818871-istio-podinfo" pod "379d09bf-6153-4be5-be07-e5ba38818871" (UID: "379d09bf-6153-4be5-be07-e5ba38818871"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:09:09.027791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:09.027751 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379d09bf-6153-4be5-be07-e5ba38818871-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:09.027791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:09.027794 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379d09bf-6153-4be5-be07-e5ba38818871-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:09.028001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:09.027808 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd6p9\" (UniqueName: \"kubernetes.io/projected/379d09bf-6153-4be5-be07-e5ba38818871-kube-api-access-rd6p9\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:09.193392 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:09.193314 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" Oct 29 23:09:09.193392 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:09.193354 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl" event={"ID":"379d09bf-6153-4be5-be07-e5ba38818871","Type":"ContainerDied","Data":"5462df73730834517cf0f9f8ebabc8219c8124247b027c1cf48c516388355b6f"} Oct 29 23:09:09.193636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:09.193467 5057 scope.go:122] "RemoveContainer" containerID="e798c22aa47acbd74128534d604ac1e9702a34370c4c9e5fb74ac5eca5667442" Oct 29 23:09:09.195226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:09.195170 5057 generic.go:356] "Generic (PLEG): container finished" podID="7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" containerID="027e0c714b6b67a7b970559717a1af03d6908446225b0b4bf54126b5cf7df370" exitCode=0 Oct 29 23:09:09.195363 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:09.195256 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" event={"ID":"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33","Type":"ContainerDied","Data":"027e0c714b6b67a7b970559717a1af03d6908446225b0b4bf54126b5cf7df370"} Oct 29 23:09:09.195603 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:09.195580 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:09.196000 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:09.195980 5057 scope.go:122] "RemoveContainer" containerID="027e0c714b6b67a7b970559717a1af03d6908446225b0b4bf54126b5cf7df370" Oct 29 23:09:09.196553 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:09.196533 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:09:09.197973 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:09.197929 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:09:09.198081 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:09.198021 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:09:09.198148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:09.198096 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:09:09.288121 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:09:09.288091 5057 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff64f87_a201_4f46_af24_e1df63c2dab7.slice/crio-14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:09:09.362047 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:09.361395 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl"] Oct 29 23:09:09.366759 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:09.366168 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-hp8nl"] Oct 29 23:09:09.484481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:09.484402 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="379d09bf-6153-4be5-be07-e5ba38818871" path="/var/lib/kubelet/pods/379d09bf-6153-4be5-be07-e5ba38818871/volumes" Oct 29 23:09:10.069225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.068793 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:09:10.171621 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.171588 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-podinfo\") pod \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " Oct 29 23:09:10.171787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.171708 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-workload-socket\") pod \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " Oct 29 23:09:10.171840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.171814 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-token\" (UniqueName: \"kubernetes.io/projected/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-token\") pod \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " Oct 29 23:09:10.171920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.171900 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istiod-ca-cert\") pod \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " Oct 29 23:09:10.171966 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.171957 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-envoy\") pod \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " Oct 29 23:09:10.172144 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.172016 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-data\") pod \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " Oct 29 23:09:10.172144 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.172085 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-kube-api-access-qdhg5\" (UniqueName: \"kubernetes.io/projected/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-kube-api-access-qdhg5\") pod \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " Oct 29 23:09:10.172309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.172156 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-workload-certs\") pod \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " Oct 29 23:09:10.172309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.172225 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-credential-socket\") pod \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\" (UID: \"fdd750a6-8487-4d03-9e52-4c6b6ef9133b\") " Oct 29 23:09:10.177838 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.175453 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-credential-socket" pod "fdd750a6-8487-4d03-9e52-4c6b6ef9133b" (UID: "fdd750a6-8487-4d03-9e52-4c6b6ef9133b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:10.177838 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.175599 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-workload-socket" pod "fdd750a6-8487-4d03-9e52-4c6b6ef9133b" (UID: "fdd750a6-8487-4d03-9e52-4c6b6ef9133b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:10.177838 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.175812 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istiod-ca-cert" pod "fdd750a6-8487-4d03-9e52-4c6b6ef9133b" (UID: "fdd750a6-8487-4d03-9e52-4c6b6ef9133b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:09:10.177838 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.176166 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-workload-certs" pod "fdd750a6-8487-4d03-9e52-4c6b6ef9133b" (UID: "fdd750a6-8487-4d03-9e52-4c6b6ef9133b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:10.177838 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.176429 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-data" pod "fdd750a6-8487-4d03-9e52-4c6b6ef9133b" (UID: "fdd750a6-8487-4d03-9e52-4c6b6ef9133b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:10.178526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.178498 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-token" pod "fdd750a6-8487-4d03-9e52-4c6b6ef9133b" (UID: "fdd750a6-8487-4d03-9e52-4c6b6ef9133b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:09:10.179613 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.179571 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-podinfo" pod "fdd750a6-8487-4d03-9e52-4c6b6ef9133b" (UID: "fdd750a6-8487-4d03-9e52-4c6b6ef9133b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:09:10.180024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.179999 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-kube-api-access-qdhg5" pod "fdd750a6-8487-4d03-9e52-4c6b6ef9133b" (UID: "fdd750a6-8487-4d03-9e52-4c6b6ef9133b"). InnerVolumeSpecName "kube-api-access-qdhg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:09:10.181297 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.181274 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-envoy" pod "fdd750a6-8487-4d03-9e52-4c6b6ef9133b" (UID: "fdd750a6-8487-4d03-9e52-4c6b6ef9133b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:10.259408 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.259366 5057 generic.go:356] "Generic (PLEG): container finished" podID="fdd750a6-8487-4d03-9e52-4c6b6ef9133b" containerID="6cd18c7f34df3865986b41baca418b6ff2c4ea914420ff8a0112a95789549aa1" exitCode=0 Oct 29 23:09:10.259636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.259426 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" Oct 29 23:09:10.259636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.259455 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" event={"ID":"fdd750a6-8487-4d03-9e52-4c6b6ef9133b","Type":"ContainerDied","Data":"6cd18c7f34df3865986b41baca418b6ff2c4ea914420ff8a0112a95789549aa1"} Oct 29 23:09:10.259636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.259527 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59" event={"ID":"fdd750a6-8487-4d03-9e52-4c6b6ef9133b","Type":"ContainerDied","Data":"801ffe65ea06927ecaa90e25c84d7ad2533def64835c6797d51517c9ce4ea1f4"} Oct 29 23:09:10.259636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.259558 5057 scope.go:122] "RemoveContainer" containerID="6cd18c7f34df3865986b41baca418b6ff2c4ea914420ff8a0112a95789549aa1" Oct 29 23:09:10.264693 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.264657 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" event={"ID":"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33","Type":"ContainerStarted","Data":"d2caf935fee120a25ff6fbc167cad2b963c5e6dd19edcd827f56bd131d21891b"} Oct 29 23:09:10.274818 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.274798 5057 scope.go:122] "RemoveContainer" containerID="d88e4d7855571e464b282f6e02800d1afca39961e0b9350fcff4a656bc6a40c4" Oct 29 23:09:10.275997 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.275974 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:10.275997 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.275996 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:10.276172 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.276005 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:10.276172 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.276014 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdhg5\" (UniqueName: \"kubernetes.io/projected/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-kube-api-access-qdhg5\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:10.276172 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.276029 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:10.276172 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.276039 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:10.276172 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.276047 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:10.276172 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.276055 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:10.276172 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.276063 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd750a6-8487-4d03-9e52-4c6b6ef9133b-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:10.285959 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.285938 5057 scope.go:122] "RemoveContainer" containerID="6cd18c7f34df3865986b41baca418b6ff2c4ea914420ff8a0112a95789549aa1" Oct 29 23:09:10.286300 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:09:10.286283 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd18c7f34df3865986b41baca418b6ff2c4ea914420ff8a0112a95789549aa1\": container with ID starting with 6cd18c7f34df3865986b41baca418b6ff2c4ea914420ff8a0112a95789549aa1 not found: ID does not exist" containerID="6cd18c7f34df3865986b41baca418b6ff2c4ea914420ff8a0112a95789549aa1" Oct 29 23:09:10.286383 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.286308 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd18c7f34df3865986b41baca418b6ff2c4ea914420ff8a0112a95789549aa1"} err="failed to get container status \"6cd18c7f34df3865986b41baca418b6ff2c4ea914420ff8a0112a95789549aa1\": rpc error: code = NotFound desc = could not find container \"6cd18c7f34df3865986b41baca418b6ff2c4ea914420ff8a0112a95789549aa1\": container with ID starting with 6cd18c7f34df3865986b41baca418b6ff2c4ea914420ff8a0112a95789549aa1 not found: ID does not exist" Oct 29 23:09:10.286383 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.286325 5057 scope.go:122] "RemoveContainer" containerID="d88e4d7855571e464b282f6e02800d1afca39961e0b9350fcff4a656bc6a40c4" Oct 29 23:09:10.286569 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:09:10.286555 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d88e4d7855571e464b282f6e02800d1afca39961e0b9350fcff4a656bc6a40c4\": container with ID starting with d88e4d7855571e464b282f6e02800d1afca39961e0b9350fcff4a656bc6a40c4 not found: ID does not exist" containerID="d88e4d7855571e464b282f6e02800d1afca39961e0b9350fcff4a656bc6a40c4" Oct 29 23:09:10.286623 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.286572 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d88e4d7855571e464b282f6e02800d1afca39961e0b9350fcff4a656bc6a40c4"} err="failed to get container status \"d88e4d7855571e464b282f6e02800d1afca39961e0b9350fcff4a656bc6a40c4\": rpc error: code = NotFound desc = could not find container \"d88e4d7855571e464b282f6e02800d1afca39961e0b9350fcff4a656bc6a40c4\": container with ID starting with d88e4d7855571e464b282f6e02800d1afca39961e0b9350fcff4a656bc6a40c4 not found: ID does not exist" Oct 29 23:09:10.401104 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.399963 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59"] Oct 29 23:09:10.414393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.414265 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-57b28e-openshift-default-6b4c4996f7-9rp59"] Oct 29 23:09:10.954209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.953317 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:09:10.954209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.953564 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 29 23:09:10.954209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:10.953626 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" podUID="7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 29 23:09:11.482908 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:11.482876 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdd750a6-8487-4d03-9e52-4c6b6ef9133b" path="/var/lib/kubelet/pods/fdd750a6-8487-4d03-9e52-4c6b6ef9133b/volumes" Oct 29 23:09:11.954211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:11.953799 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 29 23:09:11.954211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:11.953862 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" podUID="7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 29 23:09:12.558210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:12.555662 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf"] Oct 29 23:09:12.558210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:12.556838 5057 cpu_manager.go:414] "containerMap: removing container" podUID="379d09bf-6153-4be5-be07-e5ba38818871" containerName="istio-proxy" Oct 29 23:09:12.558210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:12.556857 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="379d09bf-6153-4be5-be07-e5ba38818871" containerName="istio-proxy" Oct 29 23:09:12.558210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:12.556895 5057 cpu_manager.go:414] "containerMap: removing container" podUID="fdd750a6-8487-4d03-9e52-4c6b6ef9133b" containerName="istio-proxy" Oct 29 23:09:12.558210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:12.556903 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdd750a6-8487-4d03-9e52-4c6b6ef9133b" containerName="istio-proxy" Oct 29 23:09:12.558210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:12.557157 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdd750a6-8487-4d03-9e52-4c6b6ef9133b" containerName="istio-proxy" Oct 29 23:09:12.558210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:12.557176 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="379d09bf-6153-4be5-be07-e5ba38818871" containerName="istio-proxy" Oct 29 23:09:12.558210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:12.557205 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdd750a6-8487-4d03-9e52-4c6b6ef9133b" containerName="istio-proxy" Oct 29 23:09:12.561733 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:12.561712 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf" Oct 29 23:09:12.583291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:12.583259 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf"] Oct 29 23:09:12.588441 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:12.588411 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f2c8d3\"" Oct 29 23:09:12.588580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:12.588555 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f2c8d3-dockercfg-mktwv\"" Oct 29 23:09:12.951618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:12.951580 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 29 23:09:12.951908 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:12.951872 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" podUID="7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 29 23:09:13.016747 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:13.016713 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:09:13.113213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:13.036343 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42728ab-5de3-4e1d-b388-6e5aa5e21206-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf\" (UID: \"f42728ab-5de3-4e1d-b388-6e5aa5e21206\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf" Oct 29 23:09:13.115078 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:13.115037 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg5ht\" (UniqueName: \"kubernetes.io/projected/f42728ab-5de3-4e1d-b388-6e5aa5e21206-kube-api-access-lg5ht\") pod \"payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf\" (UID: \"f42728ab-5de3-4e1d-b388-6e5aa5e21206\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf" Oct 29 23:09:13.230261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:13.217837 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42728ab-5de3-4e1d-b388-6e5aa5e21206-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf\" (UID: \"f42728ab-5de3-4e1d-b388-6e5aa5e21206\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf" Oct 29 23:09:13.244475 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:13.244439 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg5ht\" (UniqueName: \"kubernetes.io/projected/f42728ab-5de3-4e1d-b388-6e5aa5e21206-kube-api-access-lg5ht\") pod \"payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf\" (UID: \"f42728ab-5de3-4e1d-b388-6e5aa5e21206\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf" Oct 29 23:09:13.250018 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:13.232763 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42728ab-5de3-4e1d-b388-6e5aa5e21206-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf\" (UID: \"f42728ab-5de3-4e1d-b388-6e5aa5e21206\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf" Oct 29 23:09:13.548267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:13.547494 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg5ht\" (UniqueName: \"kubernetes.io/projected/f42728ab-5de3-4e1d-b388-6e5aa5e21206-kube-api-access-lg5ht\") pod \"payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf\" (UID: \"f42728ab-5de3-4e1d-b388-6e5aa5e21206\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf" Oct 29 23:09:13.794926 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:13.794891 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf" Oct 29 23:09:13.962416 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:13.962374 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 29 23:09:13.962735 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:13.962701 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" podUID="7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 29 23:09:14.868542 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:14.868330 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf"] Oct 29 23:09:14.962182 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:14.962149 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 29 23:09:14.962526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:14.962459 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" podUID="7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 29 23:09:15.667133 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:15.667094 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf" event={"ID":"f42728ab-5de3-4e1d-b388-6e5aa5e21206","Type":"ContainerStarted","Data":"b6a8f956a2a47164b9184b54740ee52ffa105983400afeec96ba78e9369a197c"} Oct 29 23:09:15.667342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:15.667159 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf" event={"ID":"f42728ab-5de3-4e1d-b388-6e5aa5e21206","Type":"ContainerStarted","Data":"da019f4da2681f45347f344218aa75732af728b2a34dc4575c05048dd3159bae"} Oct 29 23:09:15.667342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:15.667213 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf" Oct 29 23:09:15.714260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:15.714176 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf" podStartSLOduration=3.714156843 podStartE2EDuration="3.714156843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:15.711275125 +0000 UTC m=+1602.362773654" watchObservedRunningTime="2026-10-29 23:09:15.714156843 +0000 UTC m=+1602.365655375" Oct 29 23:09:15.951699 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:15.951659 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 29 23:09:15.952333 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:15.952299 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" podUID="7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 29 23:09:16.956500 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:16.956471 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:09:16.957774 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:16.957743 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:09:18.558846 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:09:18.558813 5057 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff64f87_a201_4f46_af24_e1df63c2dab7.slice/crio-14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:09:19.305103 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:09:19.305058 5057 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff64f87_a201_4f46_af24_e1df63c2dab7.slice/crio-14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:09:21.559244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:21.559168 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52"] Oct 29 23:09:21.560260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:21.560224 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" podUID="8fd55749-31a3-49ff-8965-057fb2eeb9b8" containerName="istio-proxy" containerID="cri-o://45e2ae4f8f200c2565f0f66877818df0ca5b7e12c08cdac04c600d6f45f52630" gracePeriod=30 Oct 29 23:09:22.468689 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:22.468629 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk"] Oct 29 23:09:22.469412 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:22.469177 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" podUID="83a475a5-9dd5-4278-9109-97682e026b7c" containerName="istio-proxy" containerID="cri-o://16963e219c6944a6f0cb2c5f7dcff4c6c9e06b4235acd42af69cfa6cb6bbf3fa" gracePeriod=30 Oct 29 23:09:23.947149 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:23.947113 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:09:25.384426 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:25.384391 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:09:25.385027 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:25.384994 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" podUID="8fd55749-31a3-49ff-8965-057fb2eeb9b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:09:26.650381 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:26.650342 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:09:26.650959 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:26.650926 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" podUID="83a475a5-9dd5-4278-9109-97682e026b7c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:09:26.679519 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:26.679491 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf" Oct 29 23:09:27.001757 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.001726 5057 generic.go:356] "Generic (PLEG): container finished" podID="8fd55749-31a3-49ff-8965-057fb2eeb9b8" containerID="45e2ae4f8f200c2565f0f66877818df0ca5b7e12c08cdac04c600d6f45f52630" exitCode=0 Oct 29 23:09:27.001905 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.001774 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" event={"ID":"8fd55749-31a3-49ff-8965-057fb2eeb9b8","Type":"ContainerDied","Data":"45e2ae4f8f200c2565f0f66877818df0ca5b7e12c08cdac04c600d6f45f52630"} Oct 29 23:09:27.001905 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.001823 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" event={"ID":"8fd55749-31a3-49ff-8965-057fb2eeb9b8","Type":"ContainerDied","Data":"9edc1ac63d204430079de1878674a43edd121905a33bab5bfdec2aafc345a545"} Oct 29 23:09:27.001905 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.001841 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9edc1ac63d204430079de1878674a43edd121905a33bab5bfdec2aafc345a545" Oct 29 23:09:27.001905 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.001853 5057 scope.go:122] "RemoveContainer" containerID="7a3ed1f921a2cf7022e03834d6e2feba3cf62997b2a11a5f7c073025df0c75c7" Oct 29 23:09:27.057418 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.057388 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:09:27.179212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.174169 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-workload-certs\") pod \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " Oct 29 23:09:27.179212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.174319 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-podinfo\") pod \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " Oct 29 23:09:27.179212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.174378 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-data\") pod \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " Oct 29 23:09:27.179212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.174418 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istiod-ca-cert\") pod \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " Oct 29 23:09:27.179212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.174537 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-workload-socket\") pod \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " Oct 29 23:09:27.179212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.174684 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-envoy\") pod \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " Oct 29 23:09:27.179212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.174712 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-credential-socket\") pod \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " Oct 29 23:09:27.179212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.174760 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-token\" (UniqueName: \"kubernetes.io/projected/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-token\") pod \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " Oct 29 23:09:27.179212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.174789 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fd55749-31a3-49ff-8965-057fb2eeb9b8-kube-api-access-b8clv\" (UniqueName: \"kubernetes.io/projected/8fd55749-31a3-49ff-8965-057fb2eeb9b8-kube-api-access-b8clv\") pod \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\" (UID: \"8fd55749-31a3-49ff-8965-057fb2eeb9b8\") " Oct 29 23:09:27.179836 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.179609 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istiod-ca-cert" pod "8fd55749-31a3-49ff-8965-057fb2eeb9b8" (UID: "8fd55749-31a3-49ff-8965-057fb2eeb9b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:09:27.183207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.180754 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-workload-certs" pod "8fd55749-31a3-49ff-8965-057fb2eeb9b8" (UID: "8fd55749-31a3-49ff-8965-057fb2eeb9b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:27.195233 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.193347 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-data" pod "8fd55749-31a3-49ff-8965-057fb2eeb9b8" (UID: "8fd55749-31a3-49ff-8965-057fb2eeb9b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:27.196599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.196557 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-workload-socket" pod "8fd55749-31a3-49ff-8965-057fb2eeb9b8" (UID: "8fd55749-31a3-49ff-8965-057fb2eeb9b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:27.196966 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.196924 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-credential-socket" pod "8fd55749-31a3-49ff-8965-057fb2eeb9b8" (UID: "8fd55749-31a3-49ff-8965-057fb2eeb9b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:27.258883 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.258795 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-podinfo" pod "8fd55749-31a3-49ff-8965-057fb2eeb9b8" (UID: "8fd55749-31a3-49ff-8965-057fb2eeb9b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:09:27.258883 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.258795 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-envoy" pod "8fd55749-31a3-49ff-8965-057fb2eeb9b8" (UID: "8fd55749-31a3-49ff-8965-057fb2eeb9b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:27.259080 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.258965 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd55749-31a3-49ff-8965-057fb2eeb9b8-kube-api-access-b8clv" pod "8fd55749-31a3-49ff-8965-057fb2eeb9b8" (UID: "8fd55749-31a3-49ff-8965-057fb2eeb9b8"). InnerVolumeSpecName "kube-api-access-b8clv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:09:27.260675 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.260650 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-token" pod "8fd55749-31a3-49ff-8965-057fb2eeb9b8" (UID: "8fd55749-31a3-49ff-8965-057fb2eeb9b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:09:27.277498 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.277471 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:27.277498 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.277496 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:27.277498 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.277506 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:27.277704 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.277513 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:27.277704 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.277521 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:27.277704 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.277529 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:27.277704 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.277538 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd55749-31a3-49ff-8965-057fb2eeb9b8-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:27.277704 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.277548 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8clv\" (UniqueName: \"kubernetes.io/projected/8fd55749-31a3-49ff-8965-057fb2eeb9b8-kube-api-access-b8clv\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:27.277704 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.277557 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd55749-31a3-49ff-8965-057fb2eeb9b8-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:27.816558 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.816487 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:09:27.988368 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.988325 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a475a5-9dd5-4278-9109-97682e026b7c-kube-api-access-d68rh\" (UniqueName: \"kubernetes.io/projected/83a475a5-9dd5-4278-9109-97682e026b7c-kube-api-access-d68rh\") pod \"83a475a5-9dd5-4278-9109-97682e026b7c\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " Oct 29 23:09:27.988611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.988472 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-workload-socket\") pod \"83a475a5-9dd5-4278-9109-97682e026b7c\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " Oct 29 23:09:27.988611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.988539 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a475a5-9dd5-4278-9109-97682e026b7c-istio-token\" (UniqueName: \"kubernetes.io/projected/83a475a5-9dd5-4278-9109-97682e026b7c-istio-token\") pod \"83a475a5-9dd5-4278-9109-97682e026b7c\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " Oct 29 23:09:27.988611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.988591 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83a475a5-9dd5-4278-9109-97682e026b7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a475a5-9dd5-4278-9109-97682e026b7c-istio-podinfo\") pod \"83a475a5-9dd5-4278-9109-97682e026b7c\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " Oct 29 23:09:27.988812 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.988626 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-istio-data\") pod \"83a475a5-9dd5-4278-9109-97682e026b7c\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " Oct 29 23:09:27.988812 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.988675 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a475a5-9dd5-4278-9109-97682e026b7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a475a5-9dd5-4278-9109-97682e026b7c-istiod-ca-cert\") pod \"83a475a5-9dd5-4278-9109-97682e026b7c\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " Oct 29 23:09:27.988923 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.988810 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-credential-socket\") pod \"83a475a5-9dd5-4278-9109-97682e026b7c\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " Oct 29 23:09:27.988923 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.988888 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-workload-certs\") pod \"83a475a5-9dd5-4278-9109-97682e026b7c\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " Oct 29 23:09:27.989037 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.988975 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-istio-envoy\") pod \"83a475a5-9dd5-4278-9109-97682e026b7c\" (UID: \"83a475a5-9dd5-4278-9109-97682e026b7c\") " Oct 29 23:09:27.990794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.989508 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-credential-socket" pod "83a475a5-9dd5-4278-9109-97682e026b7c" (UID: "83a475a5-9dd5-4278-9109-97682e026b7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:27.990794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.989621 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-workload-socket" pod "83a475a5-9dd5-4278-9109-97682e026b7c" (UID: "83a475a5-9dd5-4278-9109-97682e026b7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:27.990794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.989766 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:27.990794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.989788 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:27.990794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.990263 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-istio-data" pod "83a475a5-9dd5-4278-9109-97682e026b7c" (UID: "83a475a5-9dd5-4278-9109-97682e026b7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:27.990794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.990326 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-workload-certs" pod "83a475a5-9dd5-4278-9109-97682e026b7c" (UID: "83a475a5-9dd5-4278-9109-97682e026b7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:27.990794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.990760 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a475a5-9dd5-4278-9109-97682e026b7c-istiod-ca-cert" pod "83a475a5-9dd5-4278-9109-97682e026b7c" (UID: "83a475a5-9dd5-4278-9109-97682e026b7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:09:27.991711 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.991686 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a475a5-9dd5-4278-9109-97682e026b7c-kube-api-access-d68rh" pod "83a475a5-9dd5-4278-9109-97682e026b7c" (UID: "83a475a5-9dd5-4278-9109-97682e026b7c"). InnerVolumeSpecName "kube-api-access-d68rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:09:27.992167 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.992140 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-istio-envoy" pod "83a475a5-9dd5-4278-9109-97682e026b7c" (UID: "83a475a5-9dd5-4278-9109-97682e026b7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:27.992528 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.992504 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83a475a5-9dd5-4278-9109-97682e026b7c-istio-podinfo" pod "83a475a5-9dd5-4278-9109-97682e026b7c" (UID: "83a475a5-9dd5-4278-9109-97682e026b7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:09:27.993571 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:27.993546 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a475a5-9dd5-4278-9109-97682e026b7c-istio-token" pod "83a475a5-9dd5-4278-9109-97682e026b7c" (UID: "83a475a5-9dd5-4278-9109-97682e026b7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:09:28.012977 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.012947 5057 generic.go:356] "Generic (PLEG): container finished" podID="83a475a5-9dd5-4278-9109-97682e026b7c" containerID="16963e219c6944a6f0cb2c5f7dcff4c6c9e06b4235acd42af69cfa6cb6bbf3fa" exitCode=0 Oct 29 23:09:28.013137 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.013063 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" event={"ID":"83a475a5-9dd5-4278-9109-97682e026b7c","Type":"ContainerDied","Data":"16963e219c6944a6f0cb2c5f7dcff4c6c9e06b4235acd42af69cfa6cb6bbf3fa"} Oct 29 23:09:28.013137 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.013104 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" event={"ID":"83a475a5-9dd5-4278-9109-97682e026b7c","Type":"ContainerDied","Data":"48e9a3918427694787bc9eb10e88af828cbad7ea68c43fc2a2ab00ce9ad03b68"} Oct 29 23:09:28.013137 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.013124 5057 scope.go:122] "RemoveContainer" containerID="16963e219c6944a6f0cb2c5f7dcff4c6c9e06b4235acd42af69cfa6cb6bbf3fa" Oct 29 23:09:28.013260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.013242 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk" Oct 29 23:09:28.017567 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.017546 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52" Oct 29 23:09:28.063010 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.062964 5057 scope.go:122] "RemoveContainer" containerID="02c4745abefb7e2bb3424c728f63190c38ab9a773651bc308637f4ed6bc02be1" Oct 29 23:09:28.074234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.074202 5057 scope.go:122] "RemoveContainer" containerID="16963e219c6944a6f0cb2c5f7dcff4c6c9e06b4235acd42af69cfa6cb6bbf3fa" Oct 29 23:09:28.075002 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:09:28.074975 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16963e219c6944a6f0cb2c5f7dcff4c6c9e06b4235acd42af69cfa6cb6bbf3fa\": container with ID starting with 16963e219c6944a6f0cb2c5f7dcff4c6c9e06b4235acd42af69cfa6cb6bbf3fa not found: ID does not exist" containerID="16963e219c6944a6f0cb2c5f7dcff4c6c9e06b4235acd42af69cfa6cb6bbf3fa" Oct 29 23:09:28.075082 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.075014 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16963e219c6944a6f0cb2c5f7dcff4c6c9e06b4235acd42af69cfa6cb6bbf3fa"} err="failed to get container status \"16963e219c6944a6f0cb2c5f7dcff4c6c9e06b4235acd42af69cfa6cb6bbf3fa\": rpc error: code = NotFound desc = could not find container \"16963e219c6944a6f0cb2c5f7dcff4c6c9e06b4235acd42af69cfa6cb6bbf3fa\": container with ID starting with 16963e219c6944a6f0cb2c5f7dcff4c6c9e06b4235acd42af69cfa6cb6bbf3fa not found: ID does not exist" Oct 29 23:09:28.075082 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.075036 5057 scope.go:122] "RemoveContainer" containerID="02c4745abefb7e2bb3424c728f63190c38ab9a773651bc308637f4ed6bc02be1" Oct 29 23:09:28.075344 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:09:28.075323 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c4745abefb7e2bb3424c728f63190c38ab9a773651bc308637f4ed6bc02be1\": container with ID starting with 02c4745abefb7e2bb3424c728f63190c38ab9a773651bc308637f4ed6bc02be1 not found: ID does not exist" containerID="02c4745abefb7e2bb3424c728f63190c38ab9a773651bc308637f4ed6bc02be1" Oct 29 23:09:28.075382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.075355 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c4745abefb7e2bb3424c728f63190c38ab9a773651bc308637f4ed6bc02be1"} err="failed to get container status \"02c4745abefb7e2bb3424c728f63190c38ab9a773651bc308637f4ed6bc02be1\": rpc error: code = NotFound desc = could not find container \"02c4745abefb7e2bb3424c728f63190c38ab9a773651bc308637f4ed6bc02be1\": container with ID starting with 02c4745abefb7e2bb3424c728f63190c38ab9a773651bc308637f4ed6bc02be1 not found: ID does not exist" Oct 29 23:09:28.090694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.090663 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d68rh\" (UniqueName: \"kubernetes.io/projected/83a475a5-9dd5-4278-9109-97682e026b7c-kube-api-access-d68rh\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:28.090694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.090692 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a475a5-9dd5-4278-9109-97682e026b7c-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:28.090866 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.090706 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a475a5-9dd5-4278-9109-97682e026b7c-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:28.090866 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.090717 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:28.090866 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.090729 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a475a5-9dd5-4278-9109-97682e026b7c-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:28.090866 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.090741 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:28.090866 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.090753 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a475a5-9dd5-4278-9109-97682e026b7c-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:28.114077 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.114042 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk"] Oct 29 23:09:28.119274 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.119246 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-c23236-openshift-default-bc68d498f-mqzgk"] Oct 29 23:09:28.172003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.171975 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52"] Oct 29 23:09:28.177092 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:28.177068 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd000a-openshift-default-749fbf4d4f-fvp52"] Oct 29 23:09:29.348211 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:09:29.345955 5057 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff64f87_a201_4f46_af24_e1df63c2dab7.slice/crio-14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:09:29.487474 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:29.487436 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83a475a5-9dd5-4278-9109-97682e026b7c" path="/var/lib/kubelet/pods/83a475a5-9dd5-4278-9109-97682e026b7c/volumes" Oct 29 23:09:29.488001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:29.487986 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fd55749-31a3-49ff-8965-057fb2eeb9b8" path="/var/lib/kubelet/pods/8fd55749-31a3-49ff-8965-057fb2eeb9b8/volumes" Oct 29 23:09:33.460857 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:09:33.460816 5057 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff64f87_a201_4f46_af24_e1df63c2dab7.slice/crio-14ed88f37a41ad28d0fcbe98edd0a7811ef6e1e68b0602f74e61e0a3f852cd40.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:09:33.914060 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:09:33.913953 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:09:34.254030 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.253994 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm"] Oct 29 23:09:34.254765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.254748 5057 cpu_manager.go:414] "containerMap: removing container" podUID="fdd750a6-8487-4d03-9e52-4c6b6ef9133b" containerName="istio-proxy" Oct 29 23:09:34.254765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.254764 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdd750a6-8487-4d03-9e52-4c6b6ef9133b" containerName="istio-proxy" Oct 29 23:09:34.254929 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.254786 5057 cpu_manager.go:414] "containerMap: removing container" podUID="8fd55749-31a3-49ff-8965-057fb2eeb9b8" containerName="istio-proxy" Oct 29 23:09:34.254929 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.254792 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fd55749-31a3-49ff-8965-057fb2eeb9b8" containerName="istio-proxy" Oct 29 23:09:34.254929 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.254808 5057 cpu_manager.go:414] "containerMap: removing container" podUID="83a475a5-9dd5-4278-9109-97682e026b7c" containerName="istio-proxy" Oct 29 23:09:34.254929 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.254813 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a475a5-9dd5-4278-9109-97682e026b7c" containerName="istio-proxy" Oct 29 23:09:34.254929 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.254836 5057 cpu_manager.go:414] "containerMap: removing container" podUID="8fd55749-31a3-49ff-8965-057fb2eeb9b8" containerName="istio-proxy" Oct 29 23:09:34.254929 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.254841 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fd55749-31a3-49ff-8965-057fb2eeb9b8" containerName="istio-proxy" Oct 29 23:09:34.255255 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.254942 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fd55749-31a3-49ff-8965-057fb2eeb9b8" containerName="istio-proxy" Oct 29 23:09:34.255255 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.254950 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a475a5-9dd5-4278-9109-97682e026b7c" containerName="istio-proxy" Oct 29 23:09:34.255255 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.254960 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fd55749-31a3-49ff-8965-057fb2eeb9b8" containerName="istio-proxy" Oct 29 23:09:34.260359 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.260334 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" Oct 29 23:09:34.303230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.299490 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-85c702-serving-cert\"" Oct 29 23:09:34.306927 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.306899 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm"] Oct 29 23:09:34.358136 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.358100 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-maas-api-tls\") pod \"maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm\" (UID: \"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" Oct 29 23:09:34.360310 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.360276 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmrd6\" (UniqueName: \"kubernetes.io/projected/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-kube-api-access-jmrd6\") pod \"maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm\" (UID: \"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" Oct 29 23:09:34.360456 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.360388 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm\" (UID: \"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" Oct 29 23:09:34.360571 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.360550 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-metrics-certs\") pod \"maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm\" (UID: \"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" Oct 29 23:09:34.493507 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.493238 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-maas-api-tls\") pod \"maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm\" (UID: \"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" Oct 29 23:09:34.493507 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.493316 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmrd6\" (UniqueName: \"kubernetes.io/projected/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-kube-api-access-jmrd6\") pod \"maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm\" (UID: \"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" Oct 29 23:09:34.493507 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.493352 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm\" (UID: \"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" Oct 29 23:09:34.518967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.506457 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-metrics-certs\") pod \"maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm\" (UID: \"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" Oct 29 23:09:34.518967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.512280 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm\" (UID: \"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" Oct 29 23:09:34.582199 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.578956 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-metrics-certs\") pod \"maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm\" (UID: \"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" Oct 29 23:09:34.582199 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.580158 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-maas-api-tls\") pod \"maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm\" (UID: \"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" Oct 29 23:09:34.591696 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.591630 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmrd6\" (UniqueName: \"kubernetes.io/projected/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-kube-api-access-jmrd6\") pod \"maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm\" (UID: \"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" Oct 29 23:09:34.795308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.795233 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n"] Oct 29 23:09:34.882348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:34.882313 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" Oct 29 23:09:35.257713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:35.257674 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm"] Oct 29 23:09:35.260399 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:09:35.260374 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5409eeb6_ca88_4c97_9ee9_16449a4f9dd6.slice/crio-b19c47cbcd30c00ecbb1ce82b3a9bdfb3974a092e1693156d31c240658cef67d WatchSource:0}: Error finding container b19c47cbcd30c00ecbb1ce82b3a9bdfb3974a092e1693156d31c240658cef67d: Status 404 returned error can't find the container with id b19c47cbcd30c00ecbb1ce82b3a9bdfb3974a092e1693156d31c240658cef67d Oct 29 23:09:35.262443 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:35.262421 5057 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:09:36.160512 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:36.160410 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" event={"ID":"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6","Type":"ContainerStarted","Data":"11a755063c20690ca3f54eb044f81739975eb9f02702d670a26f801080425697"} Oct 29 23:09:36.160512 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:36.160478 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" event={"ID":"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6","Type":"ContainerStarted","Data":"b19c47cbcd30c00ecbb1ce82b3a9bdfb3974a092e1693156d31c240658cef67d"} Oct 29 23:09:36.160995 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:36.160732 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" Oct 29 23:09:36.162937 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:36.162911 5057 generic.go:356] "Generic (PLEG): container finished" podID="d35fd058-ef20-4d71-bd81-0ecf36bef728" containerID="9b043a9d42fbfc35ec3d5d792e10dc334e5bd73f1070aa14c95ff1e197eacbaf" exitCode=0 Oct 29 23:09:36.163076 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:36.163025 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" event={"ID":"d35fd058-ef20-4d71-bd81-0ecf36bef728","Type":"ContainerDied","Data":"9b043a9d42fbfc35ec3d5d792e10dc334e5bd73f1070aa14c95ff1e197eacbaf"} Oct 29 23:09:36.163383 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:36.163366 5057 scope.go:122] "RemoveContainer" containerID="9b043a9d42fbfc35ec3d5d792e10dc334e5bd73f1070aa14c95ff1e197eacbaf" Oct 29 23:09:36.165053 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:36.165024 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:09:36.165157 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:36.165111 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:09:36.165245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:36.165172 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:09:36.209903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:36.209856 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" podStartSLOduration=1.681732735 podStartE2EDuration="2.209842823s" totalImagesPullingTime="528.110088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:36.2061776 +0000 UTC m=+1622.857676131" watchObservedRunningTime="2026-10-29 23:09:36.209842823 +0000 UTC m=+1622.861341425" Oct 29 23:09:37.177850 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:37.177809 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" event={"ID":"d35fd058-ef20-4d71-bd81-0ecf36bef728","Type":"ContainerStarted","Data":"8effa6f953946c40884fd16bfff84c42e653ec6a0e53801f1afd1fc7e0355e9c"} Oct 29 23:09:37.614992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:37.614908 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:09:37.615155 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:37.615086 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 29 23:09:37.615155 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:37.615129 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" podUID="d35fd058-ef20-4d71-bd81-0ecf36bef728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 29 23:09:38.614402 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:38.614368 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 29 23:09:38.614846 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:38.614436 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" podUID="d35fd058-ef20-4d71-bd81-0ecf36bef728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 29 23:09:39.614582 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:39.614549 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 29 23:09:39.615014 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:39.614619 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" podUID="d35fd058-ef20-4d71-bd81-0ecf36bef728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 29 23:09:39.796457 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:39.796412 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n" podUID="74819147-4b47-42d4-8d03-c12347c309bb" containerName="payload-processing" containerID="cri-o://2240aeab2f1a3bb475a06752b52f845eef6f477998537ae701256faa40f481e2" gracePeriod=25 Oct 29 23:09:40.212031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:40.212006 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n" Oct 29 23:09:40.214738 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:40.214709 5057 generic.go:356] "Generic (PLEG): container finished" podID="74819147-4b47-42d4-8d03-c12347c309bb" containerID="2240aeab2f1a3bb475a06752b52f845eef6f477998537ae701256faa40f481e2" exitCode=0 Oct 29 23:09:40.214943 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:40.214742 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n" Oct 29 23:09:40.215126 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:40.214758 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n" event={"ID":"74819147-4b47-42d4-8d03-c12347c309bb","Type":"ContainerDied","Data":"2240aeab2f1a3bb475a06752b52f845eef6f477998537ae701256faa40f481e2"} Oct 29 23:09:40.215243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:40.215178 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n" event={"ID":"74819147-4b47-42d4-8d03-c12347c309bb","Type":"ContainerDied","Data":"cb857a29f11bec60cada3302e945c7c0accb94078ef7412aedf62e100c42dcc8"} Oct 29 23:09:40.215243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:40.215232 5057 scope.go:122] "RemoveContainer" containerID="2240aeab2f1a3bb475a06752b52f845eef6f477998537ae701256faa40f481e2" Oct 29 23:09:40.260579 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:40.260551 5057 scope.go:122] "RemoveContainer" containerID="2240aeab2f1a3bb475a06752b52f845eef6f477998537ae701256faa40f481e2" Oct 29 23:09:40.260969 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:09:40.260944 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2240aeab2f1a3bb475a06752b52f845eef6f477998537ae701256faa40f481e2\": container with ID starting with 2240aeab2f1a3bb475a06752b52f845eef6f477998537ae701256faa40f481e2 not found: ID does not exist" containerID="2240aeab2f1a3bb475a06752b52f845eef6f477998537ae701256faa40f481e2" Oct 29 23:09:40.261009 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:40.260982 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2240aeab2f1a3bb475a06752b52f845eef6f477998537ae701256faa40f481e2"} err="failed to get container status \"2240aeab2f1a3bb475a06752b52f845eef6f477998537ae701256faa40f481e2\": rpc error: code = NotFound desc = could not find container \"2240aeab2f1a3bb475a06752b52f845eef6f477998537ae701256faa40f481e2\": container with ID starting with 2240aeab2f1a3bb475a06752b52f845eef6f477998537ae701256faa40f481e2 not found: ID does not exist" Oct 29 23:09:40.403094 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:40.403062 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74819147-4b47-42d4-8d03-c12347c309bb-kube-api-access-2x7kw\" (UniqueName: \"kubernetes.io/projected/74819147-4b47-42d4-8d03-c12347c309bb-kube-api-access-2x7kw\") pod \"74819147-4b47-42d4-8d03-c12347c309bb\" (UID: \"74819147-4b47-42d4-8d03-c12347c309bb\") " Oct 29 23:09:40.403454 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:40.403430 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74819147-4b47-42d4-8d03-c12347c309bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74819147-4b47-42d4-8d03-c12347c309bb-plugins-config-volume\") pod \"74819147-4b47-42d4-8d03-c12347c309bb\" (UID: \"74819147-4b47-42d4-8d03-c12347c309bb\") " Oct 29 23:09:40.403775 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:40.403751 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74819147-4b47-42d4-8d03-c12347c309bb-plugins-config-volume" pod "74819147-4b47-42d4-8d03-c12347c309bb" (UID: "74819147-4b47-42d4-8d03-c12347c309bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:09:40.404026 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:40.404009 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74819147-4b47-42d4-8d03-c12347c309bb-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:40.405558 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:40.405538 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74819147-4b47-42d4-8d03-c12347c309bb-kube-api-access-2x7kw" pod "74819147-4b47-42d4-8d03-c12347c309bb" (UID: "74819147-4b47-42d4-8d03-c12347c309bb"). InnerVolumeSpecName "kube-api-access-2x7kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:09:40.505691 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:40.505606 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x7kw\" (UniqueName: \"kubernetes.io/projected/74819147-4b47-42d4-8d03-c12347c309bb-kube-api-access-2x7kw\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:09:40.603603 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:40.603570 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n"] Oct 29 23:09:40.614897 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:40.614865 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 29 23:09:40.615292 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:40.614916 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" podUID="d35fd058-ef20-4d71-bd81-0ecf36bef728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 29 23:09:40.618419 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:40.618394 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-31d798-58657f557d-fjj8n"] Oct 29 23:09:41.484841 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:41.484808 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74819147-4b47-42d4-8d03-c12347c309bb" path="/var/lib/kubelet/pods/74819147-4b47-42d4-8d03-c12347c309bb/volumes" Oct 29 23:09:41.614334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:41.614304 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 29 23:09:41.614508 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:41.614348 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" podUID="d35fd058-ef20-4d71-bd81-0ecf36bef728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 29 23:09:42.183958 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:42.183926 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" Oct 29 23:09:42.326252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:42.325485 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:09:42.627798 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:42.627703 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:09:43.273817 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:43.273787 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:09:47.873834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:47.873803 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx"] Oct 29 23:09:47.874551 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:47.874535 5057 cpu_manager.go:414] "containerMap: removing container" podUID="83a475a5-9dd5-4278-9109-97682e026b7c" containerName="istio-proxy" Oct 29 23:09:47.874618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:47.874552 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a475a5-9dd5-4278-9109-97682e026b7c" containerName="istio-proxy" Oct 29 23:09:47.874618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:47.874580 5057 cpu_manager.go:414] "containerMap: removing container" podUID="74819147-4b47-42d4-8d03-c12347c309bb" containerName="payload-processing" Oct 29 23:09:47.874618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:47.874585 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74819147-4b47-42d4-8d03-c12347c309bb" containerName="payload-processing" Oct 29 23:09:47.874732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:47.874721 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a475a5-9dd5-4278-9109-97682e026b7c" containerName="istio-proxy" Oct 29 23:09:47.874768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:47.874739 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="74819147-4b47-42d4-8d03-c12347c309bb" containerName="payload-processing" Oct 29 23:09:47.878655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:47.878626 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:47.913771 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:47.913737 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx"] Oct 29 23:09:47.958447 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:47.958411 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:47.958618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:47.958462 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:47.958618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:47.958530 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:47.958708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:47.958637 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:47.958708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:47.958685 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74sdl\" (UniqueName: \"kubernetes.io/projected/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-kube-api-access-74sdl\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:47.958838 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:47.958820 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:47.958886 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:47.958863 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:47.958984 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:47.958964 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:47.959054 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:47.958999 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:48.060844 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.060808 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:48.061025 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.060865 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:48.061025 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.060896 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:48.061025 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.060930 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:48.061025 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.060957 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74sdl\" (UniqueName: \"kubernetes.io/projected/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-kube-api-access-74sdl\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:48.061316 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.061046 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:48.061316 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.061090 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:48.061316 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.061143 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:48.061316 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.061170 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:48.061525 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.061408 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:48.061985 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.061609 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:48.061985 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.061788 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:48.061985 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.061937 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:48.062223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.062082 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:48.063593 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.063570 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:48.063816 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.063798 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:48.086400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.086369 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:48.086528 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.086507 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74sdl\" (UniqueName: \"kubernetes.io/projected/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-kube-api-access-74sdl\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:48.192574 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.192540 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:48.513734 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:09:48.513699 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a9c05b_ea5f_4f38_8043_a3ed71754e1a.slice/crio-cadccce3d371c87a2799e6f4cf0d0168fc499fc413990d67903dd48b383df317 WatchSource:0}: Error finding container cadccce3d371c87a2799e6f4cf0d0168fc499fc413990d67903dd48b383df317: Status 404 returned error can't find the container with id cadccce3d371c87a2799e6f4cf0d0168fc499fc413990d67903dd48b383df317 Oct 29 23:09:48.513849 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.513820 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx"] Oct 29 23:09:48.516925 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.516888 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:09:48.517019 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.516971 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:09:48.517082 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:48.517022 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:09:49.357383 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:49.357340 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" event={"ID":"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a","Type":"ContainerStarted","Data":"11a223d480b1391fee8a3d891876dc2fadbb7de2ab46b9ecb1fdca9b0ff1791e"} Oct 29 23:09:49.357896 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:49.357414 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" event={"ID":"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a","Type":"ContainerStarted","Data":"cadccce3d371c87a2799e6f4cf0d0168fc499fc413990d67903dd48b383df317"} Oct 29 23:09:49.448033 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:49.447962 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" podStartSLOduration=2.447940444 podStartE2EDuration="2.447940444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:49.42565576 +0000 UTC m=+1636.077154292" watchObservedRunningTime="2026-10-29 23:09:49.447940444 +0000 UTC m=+1636.099438975" Oct 29 23:09:50.193048 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:50.193012 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:50.195343 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:50.195312 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:09:50.195490 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:50.195381 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" podUID="f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 29 23:09:51.193337 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:51.193282 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:09:51.193742 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:51.193351 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" podUID="f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 29 23:09:52.193865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:52.193824 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:09:52.194472 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:52.193885 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" podUID="f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 29 23:09:53.193492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:53.193459 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:09:53.193675 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:53.193510 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" podUID="f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 29 23:09:54.194394 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:54.194358 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:09:54.195175 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:54.195120 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" podUID="f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 29 23:09:55.193165 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:55.193125 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:09:55.193382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:55.193235 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" podUID="f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 29 23:09:56.198050 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:56.198020 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:56.198526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:56.198506 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:56.199149 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:56.199126 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:09:59.279134 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.278872 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd"] Oct 29 23:09:59.284701 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.284675 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd" Oct 29 23:09:59.309366 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.309334 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd"] Oct 29 23:09:59.315501 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.315477 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-31d798-dockercfg-dcrbl\"" Oct 29 23:09:59.317544 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.317520 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-31d798\"" Oct 29 23:09:59.503210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.487246 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1022f1-bd79-44de-82e7-58be9da67f5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd\" (UID: \"fc1022f1-bd79-44de-82e7-58be9da67f5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd" Oct 29 23:09:59.503210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.487344 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blgh5\" (UniqueName: \"kubernetes.io/projected/fc1022f1-bd79-44de-82e7-58be9da67f5d-kube-api-access-blgh5\") pod \"payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd\" (UID: \"fc1022f1-bd79-44de-82e7-58be9da67f5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd" Oct 29 23:09:59.633277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.570367 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh"] Oct 29 23:09:59.649343 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.589156 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1022f1-bd79-44de-82e7-58be9da67f5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd\" (UID: \"fc1022f1-bd79-44de-82e7-58be9da67f5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd" Oct 29 23:09:59.649740 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.649714 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blgh5\" (UniqueName: \"kubernetes.io/projected/fc1022f1-bd79-44de-82e7-58be9da67f5d-kube-api-access-blgh5\") pod \"payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd\" (UID: \"fc1022f1-bd79-44de-82e7-58be9da67f5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd" Oct 29 23:09:59.651308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.590142 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1022f1-bd79-44de-82e7-58be9da67f5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd\" (UID: \"fc1022f1-bd79-44de-82e7-58be9da67f5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd" Oct 29 23:09:59.656094 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.651649 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh" Oct 29 23:09:59.667299 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.661303 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh"] Oct 29 23:09:59.703234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.703047 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blgh5\" (UniqueName: \"kubernetes.io/projected/fc1022f1-bd79-44de-82e7-58be9da67f5d-kube-api-access-blgh5\") pod \"payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd\" (UID: \"fc1022f1-bd79-44de-82e7-58be9da67f5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd" Oct 29 23:09:59.752408 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.752364 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59a01498-b3f7-4149-856f-5f3660ec3e94-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh\" (UID: \"59a01498-b3f7-4149-856f-5f3660ec3e94\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh" Oct 29 23:09:59.752720 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.752701 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88hqj\" (UniqueName: \"kubernetes.io/projected/59a01498-b3f7-4149-856f-5f3660ec3e94-kube-api-access-88hqj\") pod \"payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh\" (UID: \"59a01498-b3f7-4149-856f-5f3660ec3e94\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh" Oct 29 23:09:59.854215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.854161 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88hqj\" (UniqueName: \"kubernetes.io/projected/59a01498-b3f7-4149-856f-5f3660ec3e94-kube-api-access-88hqj\") pod \"payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh\" (UID: \"59a01498-b3f7-4149-856f-5f3660ec3e94\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh" Oct 29 23:09:59.854427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.854286 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59a01498-b3f7-4149-856f-5f3660ec3e94-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh\" (UID: \"59a01498-b3f7-4149-856f-5f3660ec3e94\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh" Oct 29 23:09:59.855262 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.855236 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59a01498-b3f7-4149-856f-5f3660ec3e94-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh\" (UID: \"59a01498-b3f7-4149-856f-5f3660ec3e94\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh" Oct 29 23:09:59.880441 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.880410 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88hqj\" (UniqueName: \"kubernetes.io/projected/59a01498-b3f7-4149-856f-5f3660ec3e94-kube-api-access-88hqj\") pod \"payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh\" (UID: \"59a01498-b3f7-4149-856f-5f3660ec3e94\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh" Oct 29 23:09:59.902412 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:09:59.902342 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd" Oct 29 23:10:00.013118 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:00.012901 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh" Oct 29 23:10:00.338316 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:00.338254 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd"] Oct 29 23:10:00.379768 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:10:00.379737 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a01498_b3f7_4149_856f_5f3660ec3e94.slice/crio-346a1002dcf275f3b09f491f14900103fbd72c0e85e013b4f9c89893eafe8d3d WatchSource:0}: Error finding container 346a1002dcf275f3b09f491f14900103fbd72c0e85e013b4f9c89893eafe8d3d: Status 404 returned error can't find the container with id 346a1002dcf275f3b09f491f14900103fbd72c0e85e013b4f9c89893eafe8d3d Oct 29 23:10:00.382061 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:00.382036 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh"] Oct 29 23:10:00.696200 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:00.696157 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd" event={"ID":"fc1022f1-bd79-44de-82e7-58be9da67f5d","Type":"ContainerStarted","Data":"8f64b7092aec3bb693c24d63d60e17e1f9bf101e2a9ae841948a1ef73cbcc6be"} Oct 29 23:10:00.696392 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:00.696244 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd" Oct 29 23:10:00.696392 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:00.696268 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd" event={"ID":"fc1022f1-bd79-44de-82e7-58be9da67f5d","Type":"ContainerStarted","Data":"605ca2d6749994d53a10bf1ce868b0c3dc25240cb565eb0898b6806ebe8b7a8f"} Oct 29 23:10:00.698536 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:00.698497 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh" event={"ID":"59a01498-b3f7-4149-856f-5f3660ec3e94","Type":"ContainerStarted","Data":"244c67f750b18bdf3845030acf2ed932fcef519933e5da0a2a887a2728a583bb"} Oct 29 23:10:00.698680 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:00.698556 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh" event={"ID":"59a01498-b3f7-4149-856f-5f3660ec3e94","Type":"ContainerStarted","Data":"346a1002dcf275f3b09f491f14900103fbd72c0e85e013b4f9c89893eafe8d3d"} Oct 29 23:10:00.698680 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:00.698597 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh" Oct 29 23:10:00.742480 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:00.742414 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd" podStartSLOduration=1.742396089 podStartE2EDuration="1.742396089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:00.739901434 +0000 UTC m=+1647.391399975" watchObservedRunningTime="2026-10-29 23:10:00.742396089 +0000 UTC m=+1647.393894620" Oct 29 23:10:00.790136 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:00.790070 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh" podStartSLOduration=1.790051506 podStartE2EDuration="1.790051506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:00.784441154 +0000 UTC m=+1647.435939684" watchObservedRunningTime="2026-10-29 23:10:00.790051506 +0000 UTC m=+1647.441550033" Oct 29 23:10:02.883212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:02.879942 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:10:02.982211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:02.981929 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf"] Oct 29 23:10:03.020787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.020755 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.039950 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.039916 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf"] Oct 29 23:10:03.131389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.130707 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.131389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.130807 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.131389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.130854 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.131389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.130988 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.131389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.131029 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.131389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.131074 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.131389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.131093 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.131389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.131148 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.131389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.131224 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwdbj\" (UniqueName: \"kubernetes.io/projected/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-kube-api-access-xwdbj\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.232957 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.232913 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.233168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.232982 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.233168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.233077 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.233168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.233151 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.233377 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.233223 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.233377 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.233249 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.233377 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.233316 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.233377 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.233359 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwdbj\" (UniqueName: \"kubernetes.io/projected/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-kube-api-access-xwdbj\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.233596 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.233435 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.233893 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.233867 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.234409 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.234381 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.234678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.234647 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.234802 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.234617 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.234802 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.234431 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.235932 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.235904 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.236663 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.236638 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.260077 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.260039 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwdbj\" (UniqueName: \"kubernetes.io/projected/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-kube-api-access-xwdbj\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.260534 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.260509 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.355791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.355744 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:03.711238 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:10:03.711174 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea22168_c52a_43f3_a4b6_4cdd87cffdae.slice/crio-a3a7eea2a268a5aaa1d5da4573466a51a593973a2174197d664f2e071ec6eaa7 WatchSource:0}: Error finding container a3a7eea2a268a5aaa1d5da4573466a51a593973a2174197d664f2e071ec6eaa7: Status 404 returned error can't find the container with id a3a7eea2a268a5aaa1d5da4573466a51a593973a2174197d664f2e071ec6eaa7 Oct 29 23:10:03.712628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.712597 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf"] Oct 29 23:10:03.720256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.720205 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:03.720365 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.720304 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:03.720365 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.720354 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:03.725225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:03.725164 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" event={"ID":"7ea22168-c52a-43f3-a4b6-4cdd87cffdae","Type":"ContainerStarted","Data":"a3a7eea2a268a5aaa1d5da4573466a51a593973a2174197d664f2e071ec6eaa7"} Oct 29 23:10:04.008262 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:10:04.001746 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:10:04.774213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:04.769144 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" event={"ID":"7ea22168-c52a-43f3-a4b6-4cdd87cffdae","Type":"ContainerStarted","Data":"bf1123dde721ceff0320388d9b207cd4547212466dd22e13ddf134c9ca522cde"} Oct 29 23:10:05.361246 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:05.360709 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:05.361246 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:05.360899 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 29 23:10:05.361246 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:05.360950 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" podUID="7ea22168-c52a-43f3-a4b6-4cdd87cffdae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 29 23:10:06.358653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:06.358601 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 29 23:10:06.358970 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:06.358944 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" podUID="7ea22168-c52a-43f3-a4b6-4cdd87cffdae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 29 23:10:06.574347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:06.574292 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:06.732212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:06.697443 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:07.101271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:07.095984 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" podStartSLOduration=5.095964374 podStartE2EDuration="5.095964374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:04.854573599 +0000 UTC m=+1651.506072130" watchObservedRunningTime="2026-10-29 23:10:07.095964374 +0000 UTC m=+1653.747462907" Oct 29 23:10:07.132765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:07.127119 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j"] Oct 29 23:10:07.132765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:07.127374 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" podUID="5ac55fce-ddaa-4ffc-902d-07a374f6d967" containerName="istio-proxy" containerID="cri-o://f91b03e21921908c78edd1c4fb4eac34c120b38a8dfedf5ace832ad296518175" gracePeriod=30 Oct 29 23:10:07.408273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:07.372371 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:07.408273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:07.402460 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 29 23:10:07.408273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:07.402519 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" podUID="7ea22168-c52a-43f3-a4b6-4cdd87cffdae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 29 23:10:07.639211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:07.638094 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:07.710725 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:07.653041 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:07.710725 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:07.708368 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:07.826212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:07.823523 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:10:07.826212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:07.823600 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:10:08.449244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:08.394774 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:10:08.449244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:08.395389 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:10:08.453209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:08.451775 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 29 23:10:08.453209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:08.451837 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" podUID="7ea22168-c52a-43f3-a4b6-4cdd87cffdae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 29 23:10:08.677215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:08.639368 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 23:10:08.738269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:08.669457 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 23:10:08.772659 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:08.745791 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:10:08.772659 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:08.709987 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:09.400692 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:09.400636 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:09.536628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:09.528825 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 29 23:10:09.536628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:09.528896 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" podUID="7ea22168-c52a-43f3-a4b6-4cdd87cffdae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 29 23:10:09.733965 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:09.733917 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:09.828224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:09.826964 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 23:10:09.828224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:09.827299 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:10.369948 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:10.369503 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 29 23:10:10.369948 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:10.369575 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" podUID="7ea22168-c52a-43f3-a4b6-4cdd87cffdae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 29 23:10:10.405868 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:10.404997 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:10.647268 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:10.646168 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded" Oct 29 23:10:10.734070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:10.731895 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:10:10.734070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:10.733275 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:10:10.744832 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:10.744257 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:10.839942 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:10.838339 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:11.023249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:11.018300 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:10:11.033209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:11.031473 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:10:11.420220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:11.393535 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 29 23:10:11.420220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:11.393597 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" podUID="7ea22168-c52a-43f3-a4b6-4cdd87cffdae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 29 23:10:11.751217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:11.743340 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh" Oct 29 23:10:11.751217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:11.744632 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd" Oct 29 23:10:11.839428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:11.839386 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:12.357163 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:12.356700 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 29 23:10:12.357163 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:12.356766 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" podUID="7ea22168-c52a-43f3-a4b6-4cdd87cffdae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 29 23:10:12.717658 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:12.717458 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v_cdf4f520-c1fb-4178-9685-2ddf12262613/payload-processing/0.log" Oct 29 23:10:12.814210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:12.813347 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:12.853374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:12.843071 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:12.853374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:12.843624 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:10:13.409108 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.409050 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:10:13.414902 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.414869 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 29 23:10:13.415027 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.414926 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" podUID="7ea22168-c52a-43f3-a4b6-4cdd87cffdae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 29 23:10:13.454105 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.433471 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd_e5272969-d30b-4e39-93b3-587d000ef056/payload-processing/0.log" Oct 29 23:10:13.565475 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.565416 5057 generic.go:356] "Generic (PLEG): container finished" podID="5ac55fce-ddaa-4ffc-902d-07a374f6d967" containerID="f91b03e21921908c78edd1c4fb4eac34c120b38a8dfedf5ace832ad296518175" exitCode=0 Oct 29 23:10:13.615213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.612837 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 23:10:13.648650 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.648618 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd_e5272969-d30b-4e39-93b3-587d000ef056/payload-processing/0.log" Oct 29 23:10:13.687373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.687341 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-workload-certs\") pod \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " Oct 29 23:10:13.717343 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.717267 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-workload-certs" pod "5ac55fce-ddaa-4ffc-902d-07a374f6d967" (UID: "5ac55fce-ddaa-4ffc-902d-07a374f6d967"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:13.717582 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.717366 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac55fce-ddaa-4ffc-902d-07a374f6d967-kube-api-access-6bswd\" (UniqueName: \"kubernetes.io/projected/5ac55fce-ddaa-4ffc-902d-07a374f6d967-kube-api-access-6bswd\") pod \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " Oct 29 23:10:13.717732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.717709 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-workload-socket\") pod \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " Oct 29 23:10:13.717921 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.717868 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-credential-socket\") pod \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " Oct 29 23:10:13.718248 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.718218 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-podinfo\") pod \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " Oct 29 23:10:13.718455 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.718438 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-token\" (UniqueName: \"kubernetes.io/projected/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-token\") pod \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " Oct 29 23:10:13.718699 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.718680 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istiod-ca-cert\") pod \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " Oct 29 23:10:13.718906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.718889 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-data\") pod \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " Oct 29 23:10:13.719390 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.719367 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-envoy\") pod \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\" (UID: \"5ac55fce-ddaa-4ffc-902d-07a374f6d967\") " Oct 29 23:10:13.720704 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.720683 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:13.727669 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.727642 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istiod-ca-cert" pod "5ac55fce-ddaa-4ffc-902d-07a374f6d967" (UID: "5ac55fce-ddaa-4ffc-902d-07a374f6d967"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:13.745202 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.745144 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-credential-socket" pod "5ac55fce-ddaa-4ffc-902d-07a374f6d967" (UID: "5ac55fce-ddaa-4ffc-902d-07a374f6d967"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:13.745587 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.745564 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-data" pod "5ac55fce-ddaa-4ffc-902d-07a374f6d967" (UID: "5ac55fce-ddaa-4ffc-902d-07a374f6d967"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:13.816053 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.816004 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-workload-socket" pod "5ac55fce-ddaa-4ffc-902d-07a374f6d967" (UID: "5ac55fce-ddaa-4ffc-902d-07a374f6d967"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:13.818400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.818364 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-envoy" pod "5ac55fce-ddaa-4ffc-902d-07a374f6d967" (UID: "5ac55fce-ddaa-4ffc-902d-07a374f6d967"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:13.818769 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.818738 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" event={"ID":"5ac55fce-ddaa-4ffc-902d-07a374f6d967","Type":"ContainerDied","Data":"f91b03e21921908c78edd1c4fb4eac34c120b38a8dfedf5ace832ad296518175"} Oct 29 23:10:13.819014 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.818989 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:10:13.820421 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.820398 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac55fce-ddaa-4ffc-902d-07a374f6d967-kube-api-access-6bswd" pod "5ac55fce-ddaa-4ffc-902d-07a374f6d967" (UID: "5ac55fce-ddaa-4ffc-902d-07a374f6d967"). InnerVolumeSpecName "kube-api-access-6bswd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:13.820570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.820555 5057 scope.go:122] "RemoveContainer" containerID="f91b03e21921908c78edd1c4fb4eac34c120b38a8dfedf5ace832ad296518175" Oct 29 23:10:13.842802 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.842771 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bswd\" (UniqueName: \"kubernetes.io/projected/5ac55fce-ddaa-4ffc-902d-07a374f6d967-kube-api-access-6bswd\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:13.939165 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.939081 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:13.939475 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.939456 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:13.939580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.939569 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:13.939687 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.939678 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:13.939784 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.939774 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:13.939872 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:13.866321 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-token" pod "5ac55fce-ddaa-4ffc-902d-07a374f6d967" (UID: "5ac55fce-ddaa-4ffc-902d-07a374f6d967"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:14.045365 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:14.045328 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-podinfo" pod "5ac55fce-ddaa-4ffc-902d-07a374f6d967" (UID: "5ac55fce-ddaa-4ffc-902d-07a374f6d967"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:10:14.109287 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:14.104332 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:14.109552 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:14.109533 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ac55fce-ddaa-4ffc-902d-07a374f6d967-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:14.369870 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:14.369828 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 29 23:10:14.370169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:14.370135 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" podUID="7ea22168-c52a-43f3-a4b6-4cdd87cffdae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 29 23:10:14.595955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:14.595915 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" event={"ID":"5ac55fce-ddaa-4ffc-902d-07a374f6d967","Type":"ContainerDied","Data":"83b86b0d2cad2b4f514ece83f52207780f39a14649b322532670070bf68ffe45"} Oct 29 23:10:14.596171 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:14.596021 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j" Oct 29 23:10:14.657467 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:14.657433 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j"] Oct 29 23:10:14.670971 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:14.670937 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-g5c2j"] Oct 29 23:10:15.356983 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:15.356945 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 29 23:10:15.357487 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:15.357011 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" podUID="7ea22168-c52a-43f3-a4b6-4cdd87cffdae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 29 23:10:15.488766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:15.488721 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ac55fce-ddaa-4ffc-902d-07a374f6d967" path="/var/lib/kubelet/pods/5ac55fce-ddaa-4ffc-902d-07a374f6d967/volumes" Oct 29 23:10:16.362385 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:16.362165 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:16.363768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:16.363747 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:16.364485 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:16.364463 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:10:19.667051 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:19.667013 5057 generic.go:356] "Generic (PLEG): container finished" podID="82365a6f-eb57-45d8-bcd4-ff4e8daddf14" containerID="760b92627049a7308e812ccd98760a670aeb09ff04746396f39b74ca578a090f" exitCode=0 Oct 29 23:10:19.667657 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:19.667093 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" event={"ID":"82365a6f-eb57-45d8-bcd4-ff4e8daddf14","Type":"ContainerDied","Data":"760b92627049a7308e812ccd98760a670aeb09ff04746396f39b74ca578a090f"} Oct 29 23:10:19.667657 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:19.667648 5057 scope.go:122] "RemoveContainer" containerID="760b92627049a7308e812ccd98760a670aeb09ff04746396f39b74ca578a090f" Oct 29 23:10:19.669319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:19.669296 5057 generic.go:356] "Generic (PLEG): container finished" podID="72b4ea44-8863-40e5-baee-a5e1e0c94285" containerID="9f0fd9f5b13f83680835610c4c9f4b45f72d05c529fff22ee16a2fdc83456df4" exitCode=0 Oct 29 23:10:19.669454 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:19.669342 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" event={"ID":"72b4ea44-8863-40e5-baee-a5e1e0c94285","Type":"ContainerDied","Data":"9f0fd9f5b13f83680835610c4c9f4b45f72d05c529fff22ee16a2fdc83456df4"} Oct 29 23:10:19.669693 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:19.669675 5057 scope.go:122] "RemoveContainer" containerID="9f0fd9f5b13f83680835610c4c9f4b45f72d05c529fff22ee16a2fdc83456df4" Oct 29 23:10:19.669776 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:19.669717 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:19.669854 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:19.669834 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:19.669911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:19.669891 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:19.671246 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:19.671218 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:19.671348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:19.671277 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:19.671348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:19.671309 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:20.443088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:20.443045 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc"] Oct 29 23:10:20.471269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:20.471233 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t"] Oct 29 23:10:20.720043 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:20.718552 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" event={"ID":"82365a6f-eb57-45d8-bcd4-ff4e8daddf14","Type":"ContainerStarted","Data":"ed6cc162e1ddbd26cdb05d91cdd09f96533c2a1adc6d3821a53539fc053a32a0"} Oct 29 23:10:20.735209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:20.731433 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" event={"ID":"72b4ea44-8863-40e5-baee-a5e1e0c94285","Type":"ContainerStarted","Data":"ae3e383566ea7c6451ef12660a6ae7e23e8e30e2ddb69b02bfe85dd52471b02b"} Oct 29 23:10:21.075265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:21.057296 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:10:21.290474 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:21.290441 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:10:21.290845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:21.290822 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 29 23:10:21.290994 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:21.290971 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" podUID="72b4ea44-8863-40e5-baee-a5e1e0c94285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 29 23:10:21.456217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:21.455539 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:10:21.456217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:21.455696 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 29 23:10:21.456217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:21.455753 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" podUID="82365a6f-eb57-45d8-bcd4-ff4e8daddf14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 29 23:10:22.287906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:22.287549 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 29 23:10:22.287906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:22.287616 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" podUID="72b4ea44-8863-40e5-baee-a5e1e0c94285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 29 23:10:22.446949 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:22.446911 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 29 23:10:22.449412 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:22.447309 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" podUID="82365a6f-eb57-45d8-bcd4-ff4e8daddf14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 29 23:10:23.287728 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:23.287681 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 29 23:10:23.287955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:23.287748 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" podUID="72b4ea44-8863-40e5-baee-a5e1e0c94285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 29 23:10:23.451210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:23.450631 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 29 23:10:23.451210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:23.450693 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" podUID="82365a6f-eb57-45d8-bcd4-ff4e8daddf14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 29 23:10:24.234448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.234411 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:10:24.357212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.289519 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 29 23:10:24.357212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.289597 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" podUID="72b4ea44-8863-40e5-baee-a5e1e0c94285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 29 23:10:24.473914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.472453 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 29 23:10:24.473914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.472549 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" podUID="82365a6f-eb57-45d8-bcd4-ff4e8daddf14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 29 23:10:24.491258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.487447 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr"] Oct 29 23:10:24.581261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.558728 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5ac55fce-ddaa-4ffc-902d-07a374f6d967" containerName="istio-proxy" Oct 29 23:10:24.612689 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.612655 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac55fce-ddaa-4ffc-902d-07a374f6d967" containerName="istio-proxy" Oct 29 23:10:24.612998 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.612982 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac55fce-ddaa-4ffc-902d-07a374f6d967" containerName="istio-proxy" Oct 29 23:10:24.618723 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.618697 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.619933 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.618582 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl"] Oct 29 23:10:24.691600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.690912 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.691600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.690975 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.691600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.691029 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.691600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.691089 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.691600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.691128 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.691600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.691168 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.691600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.691219 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.691600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.691277 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx8ww\" (UniqueName: \"kubernetes.io/projected/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-kube-api-access-bx8ww\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.691600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.691311 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.697223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.697169 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr"] Oct 29 23:10:24.697381 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.697322 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.700763 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.700742 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl"] Oct 29 23:10:24.794088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.793999 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.794088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.794054 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.794088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.794082 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.794365 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.794134 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.794365 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.794156 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.794866 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.794842 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.794968 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.794944 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.795013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.795002 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.795088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.795071 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.795141 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.795128 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.795204 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.795169 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.795263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.795218 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.795263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.795253 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-626k9\" (UniqueName: \"kubernetes.io/projected/b5d56fdf-4c0d-4243-980d-ae3e958544c2-kube-api-access-626k9\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.795459 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.795419 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.795530 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.795461 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.795530 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.795509 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx8ww\" (UniqueName: \"kubernetes.io/projected/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-kube-api-access-bx8ww\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.795649 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.795550 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.795649 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.795554 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.795649 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.795584 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.795764 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.795666 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.795971 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.795949 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.796434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.796387 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.796554 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.796458 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.798042 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.798020 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.799644 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.799626 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.921219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.898230 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.921219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.898402 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.921219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.898480 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.921219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.898573 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.921219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.898648 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-626k9\" (UniqueName: \"kubernetes.io/projected/b5d56fdf-4c0d-4243-980d-ae3e958544c2-kube-api-access-626k9\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.921219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.898686 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.921219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.898714 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.921219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.898797 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.921219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.898861 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.921219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.899880 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.921219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.900124 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:24.921219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.900435 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.921219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.900885 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.921219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.901116 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.928449 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.928419 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.936154 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.936081 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:24.987211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.982746 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx8ww\" (UniqueName: \"kubernetes.io/projected/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-kube-api-access-bx8ww\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:25.001210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:24.987413 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:25.003024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.002956 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-626k9\" (UniqueName: \"kubernetes.io/projected/b5d56fdf-4c0d-4243-980d-ae3e958544c2-kube-api-access-626k9\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:25.003630 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.003603 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:25.009594 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.009564 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:25.017612 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.017592 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:25.291039 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.290601 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 29 23:10:25.291039 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.290677 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" podUID="72b4ea44-8863-40e5-baee-a5e1e0c94285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 29 23:10:25.452106 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.451307 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc" podUID="272e697e-08f1-44a5-838b-742a0f570253" containerName="payload-processing" containerID="cri-o://a621481b6a8bd3c554b67fdb33f63b6c48b6c5db0d32a080eda91f5c27d7e63d" gracePeriod=25 Oct 29 23:10:25.454390 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.453877 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 29 23:10:25.454390 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.453931 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" podUID="82365a6f-eb57-45d8-bcd4-ff4e8daddf14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 29 23:10:25.546794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.505436 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t" podUID="9bb12844-8806-48b3-af64-ec85f9f77eec" containerName="payload-pre-processing" containerID="cri-o://c8d19fff5b0e03cc1d89b822139f5314ba74ef94e650aeeebfc43cddb9ac1af6" gracePeriod=25 Oct 29 23:10:25.713217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.712451 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr"] Oct 29 23:10:25.726212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.721316 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl"] Oct 29 23:10:25.795503 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.795464 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:25.795696 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.795679 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:25.795833 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.795816 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:25.798235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.798158 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:25.798499 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.798467 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:25.798609 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.798581 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:25.901156 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.901122 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" event={"ID":"b5d56fdf-4c0d-4243-980d-ae3e958544c2","Type":"ContainerStarted","Data":"58265bed9b73c4c7026c0be4ab96512afc4ed2ba62372e1d93de597f71d2e18a"} Oct 29 23:10:25.903160 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.903099 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" event={"ID":"bff30ab0-9c2d-41f0-902f-685ae8c67c0a","Type":"ContainerStarted","Data":"02b07883a15980d1c259fce9983aa3c7c51e146ccaf460f6187dd3743a408231"} Oct 29 23:10:25.905178 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.905003 5057 generic.go:356] "Generic (PLEG): container finished" podID="272e697e-08f1-44a5-838b-742a0f570253" containerID="a621481b6a8bd3c554b67fdb33f63b6c48b6c5db0d32a080eda91f5c27d7e63d" exitCode=0 Oct 29 23:10:25.905178 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.905111 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc" event={"ID":"272e697e-08f1-44a5-838b-742a0f570253","Type":"ContainerDied","Data":"a621481b6a8bd3c554b67fdb33f63b6c48b6c5db0d32a080eda91f5c27d7e63d"} Oct 29 23:10:25.908229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.908147 5057 generic.go:356] "Generic (PLEG): container finished" podID="9bb12844-8806-48b3-af64-ec85f9f77eec" containerID="c8d19fff5b0e03cc1d89b822139f5314ba74ef94e650aeeebfc43cddb9ac1af6" exitCode=0 Oct 29 23:10:25.908462 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:25.908397 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t" event={"ID":"9bb12844-8806-48b3-af64-ec85f9f77eec","Type":"ContainerDied","Data":"c8d19fff5b0e03cc1d89b822139f5314ba74ef94e650aeeebfc43cddb9ac1af6"} Oct 29 23:10:26.011545 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.011517 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9"] Oct 29 23:10:26.032764 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.032739 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.174664 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.126546 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9"] Oct 29 23:10:26.213282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.212654 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.213282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.212702 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.213282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.212766 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.213282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.212838 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67wkh\" (UniqueName: \"kubernetes.io/projected/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-kube-api-access-67wkh\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.213282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.212902 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.213282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.212954 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.213282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.212994 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.213282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.213159 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.213282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.213231 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.308676 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.308626 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 29 23:10:26.308854 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.308682 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" podUID="72b4ea44-8863-40e5-baee-a5e1e0c94285" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 29 23:10:26.319679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.317530 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc" Oct 29 23:10:26.347035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.320814 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/272e697e-08f1-44a5-838b-742a0f570253-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272e697e-08f1-44a5-838b-742a0f570253-plugins-config-volume\") pod \"272e697e-08f1-44a5-838b-742a0f570253\" (UID: \"272e697e-08f1-44a5-838b-742a0f570253\") " Oct 29 23:10:26.347035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.320910 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/272e697e-08f1-44a5-838b-742a0f570253-kube-api-access-n5xf6\" (UniqueName: \"kubernetes.io/projected/272e697e-08f1-44a5-838b-742a0f570253-kube-api-access-n5xf6\") pod \"272e697e-08f1-44a5-838b-742a0f570253\" (UID: \"272e697e-08f1-44a5-838b-742a0f570253\") " Oct 29 23:10:26.347035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.321254 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.347035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.321318 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.347035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.321432 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.347035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.321466 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.347035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.321533 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.347035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.321598 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67wkh\" (UniqueName: \"kubernetes.io/projected/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-kube-api-access-67wkh\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.347035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.321662 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.347035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.321700 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.347035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.321740 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.347035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.322323 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.347035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.332442 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272e697e-08f1-44a5-838b-742a0f570253-plugins-config-volume" pod "272e697e-08f1-44a5-838b-742a0f570253" (UID: "272e697e-08f1-44a5-838b-742a0f570253"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:26.347035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.333453 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.347035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.334082 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.530304 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.472405 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272e697e-08f1-44a5-838b-742a0f570253-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:26.687103 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.480856 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.687103 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.483309 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.687103 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.684565 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.687103 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.684881 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 29 23:10:26.687103 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.686750 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" podUID="82365a6f-eb57-45d8-bcd4-ff4e8daddf14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 29 23:10:26.720513 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.720450 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272e697e-08f1-44a5-838b-742a0f570253-kube-api-access-n5xf6" pod "272e697e-08f1-44a5-838b-742a0f570253" (UID: "272e697e-08f1-44a5-838b-742a0f570253"). InnerVolumeSpecName "kube-api-access-n5xf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:26.730941 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.730873 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.817648 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.817528 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67wkh\" (UniqueName: \"kubernetes.io/projected/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-kube-api-access-67wkh\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.818094 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.818065 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5xf6\" (UniqueName: \"kubernetes.io/projected/272e697e-08f1-44a5-838b-742a0f570253-kube-api-access-n5xf6\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:26.893255 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.876502 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:26.997264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.994366 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc" event={"ID":"272e697e-08f1-44a5-838b-742a0f570253","Type":"ContainerDied","Data":"4d3e3f4875de5b8f04b8a2034ad23966f80c1cc3ef2081e05e3f4fda6b0a67d6"} Oct 29 23:10:26.997264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.994442 5057 scope.go:122] "RemoveContainer" containerID="a621481b6a8bd3c554b67fdb33f63b6c48b6c5db0d32a080eda91f5c27d7e63d" Oct 29 23:10:26.997264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:26.994627 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc" Oct 29 23:10:27.002295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.002179 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t" Oct 29 23:10:27.005551 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.005516 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t" event={"ID":"9bb12844-8806-48b3-af64-ec85f9f77eec","Type":"ContainerDied","Data":"7cf7b08a6d29cdd290919af7d7ab1481c98dd1fc010e95dd90d510ad7667ea95"} Oct 29 23:10:27.095994 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.089111 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" event={"ID":"b5d56fdf-4c0d-4243-980d-ae3e958544c2","Type":"ContainerStarted","Data":"ee07dcf24644b423262a3c8f4d06aae1c9fc6cb4bfd329333e8ddc8fa57e7ae6"} Oct 29 23:10:27.105555 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.105507 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" event={"ID":"bff30ab0-9c2d-41f0-902f-685ae8c67c0a","Type":"ContainerStarted","Data":"2c4a58c98e1ddb110e7aae38ac189b7bdf9be0b36b0d4f6bc8e46138fcf8220c"} Oct 29 23:10:27.114167 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.113809 5057 scope.go:122] "RemoveContainer" containerID="c8d19fff5b0e03cc1d89b822139f5314ba74ef94e650aeeebfc43cddb9ac1af6" Oct 29 23:10:27.115487 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.115461 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:27.162510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.160306 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb12844-8806-48b3-af64-ec85f9f77eec-kube-api-access-mrmd2\" (UniqueName: \"kubernetes.io/projected/9bb12844-8806-48b3-af64-ec85f9f77eec-kube-api-access-mrmd2\") pod \"9bb12844-8806-48b3-af64-ec85f9f77eec\" (UID: \"9bb12844-8806-48b3-af64-ec85f9f77eec\") " Oct 29 23:10:27.162510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.160360 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb12844-8806-48b3-af64-ec85f9f77eec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb12844-8806-48b3-af64-ec85f9f77eec-plugins-config-volume\") pod \"9bb12844-8806-48b3-af64-ec85f9f77eec\" (UID: \"9bb12844-8806-48b3-af64-ec85f9f77eec\") " Oct 29 23:10:27.162510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.162146 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb12844-8806-48b3-af64-ec85f9f77eec-plugins-config-volume" pod "9bb12844-8806-48b3-af64-ec85f9f77eec" (UID: "9bb12844-8806-48b3-af64-ec85f9f77eec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:27.205752 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.189488 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc"] Oct 29 23:10:27.205752 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.199553 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6d21fd-55c469f77d-96bjc"] Oct 29 23:10:27.205752 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.200899 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" podStartSLOduration=3.200878835 podStartE2EDuration="3.200878835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:27.192132178 +0000 UTC m=+1673.843630749" watchObservedRunningTime="2026-10-29 23:10:27.200878835 +0000 UTC m=+1673.852377366" Oct 29 23:10:27.205752 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.204419 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb12844-8806-48b3-af64-ec85f9f77eec-kube-api-access-mrmd2" pod "9bb12844-8806-48b3-af64-ec85f9f77eec" (UID: "9bb12844-8806-48b3-af64-ec85f9f77eec"). InnerVolumeSpecName "kube-api-access-mrmd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:27.249011 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.248939 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" podStartSLOduration=3.248902222 podStartE2EDuration="3.248902222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:27.248782653 +0000 UTC m=+1673.900281184" watchObservedRunningTime="2026-10-29 23:10:27.248902222 +0000 UTC m=+1673.900400752" Oct 29 23:10:27.263526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.263492 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrmd2\" (UniqueName: \"kubernetes.io/projected/9bb12844-8806-48b3-af64-ec85f9f77eec-kube-api-access-mrmd2\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:27.263526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.263527 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb12844-8806-48b3-af64-ec85f9f77eec-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:27.383434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.382923 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:10:27.386525 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.386486 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:10:27.451406 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.451376 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:10:27.452621 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.452596 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:10:27.513342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.513304 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="272e697e-08f1-44a5-838b-742a0f570253" path="/var/lib/kubelet/pods/272e697e-08f1-44a5-838b-742a0f570253/volumes" Oct 29 23:10:27.598361 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.598329 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9"] Oct 29 23:10:27.789298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.788493 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:27.789298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.788583 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:27.789298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:27.788627 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:28.010124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:28.010087 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:28.011852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:28.011821 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 29 23:10:28.012006 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:28.011883 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" podUID="bff30ab0-9c2d-41f0-902f-685ae8c67c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 29 23:10:28.018177 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:28.018155 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:28.019254 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:28.019231 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 29 23:10:28.019323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:28.019271 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" podUID="b5d56fdf-4c0d-4243-980d-ae3e958544c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 29 23:10:28.116173 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:28.116140 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" event={"ID":"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4","Type":"ContainerStarted","Data":"c7309b2929fd6e102fa50ad5d34ff54ecfafc11b388ed5440de25dabf06e8e72"} Oct 29 23:10:28.116373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:28.116205 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" event={"ID":"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4","Type":"ContainerStarted","Data":"7f2864fcb92c8ae887de76cb3928e5fce0f3956fd3aa883bc05c67d86ce13a04"} Oct 29 23:10:28.118173 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:28.118152 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t" Oct 29 23:10:28.212210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:28.211676 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" podStartSLOduration=3.211656664 podStartE2EDuration="3.211656664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:28.203521603 +0000 UTC m=+1674.855020134" watchObservedRunningTime="2026-10-29 23:10:28.211656664 +0000 UTC m=+1674.863155195" Oct 29 23:10:28.303202 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:28.303140 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t"] Oct 29 23:10:28.303408 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:28.303218 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6d21fd-758f4946bc-9ss9t"] Oct 29 23:10:29.010019 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:29.009981 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 29 23:10:29.010453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:29.010035 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" podUID="bff30ab0-9c2d-41f0-902f-685ae8c67c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 29 23:10:29.018531 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:29.018501 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 29 23:10:29.018667 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:29.018570 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" podUID="b5d56fdf-4c0d-4243-980d-ae3e958544c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 29 23:10:29.116269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:29.116237 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:29.117506 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:29.117474 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 29 23:10:29.117630 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:29.117531 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" podUID="a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 29 23:10:29.724220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:29.716703 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb12844-8806-48b3-af64-ec85f9f77eec" path="/var/lib/kubelet/pods/9bb12844-8806-48b3-af64-ec85f9f77eec/volumes" Oct 29 23:10:30.058273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:30.018399 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 29 23:10:30.058273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:30.018465 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" podUID="b5d56fdf-4c0d-4243-980d-ae3e958544c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 29 23:10:30.058273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:30.018557 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 29 23:10:30.058273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:30.018600 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" podUID="bff30ab0-9c2d-41f0-902f-685ae8c67c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 29 23:10:30.121853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:30.117294 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 29 23:10:30.121853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:30.117357 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" podUID="a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 29 23:10:30.541699 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:30.541390 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:30.990892 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:30.979370 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 23:10:31.062732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:31.062688 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 29 23:10:31.063249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:31.062799 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" podUID="bff30ab0-9c2d-41f0-902f-685ae8c67c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 29 23:10:31.063828 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:31.063797 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 29 23:10:31.063937 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:31.063849 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" podUID="b5d56fdf-4c0d-4243-980d-ae3e958544c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 29 23:10:31.146262 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:31.126569 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 29 23:10:31.146262 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:31.126664 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" podUID="a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 29 23:10:32.059213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:32.024280 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 29 23:10:32.106208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:32.034810 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 29 23:10:32.106208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:32.075515 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" podUID="b5d56fdf-4c0d-4243-980d-ae3e958544c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 29 23:10:32.140246 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:32.129891 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" podUID="bff30ab0-9c2d-41f0-902f-685ae8c67c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 29 23:10:32.201129 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:32.133947 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 29 23:10:32.201129 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:32.200440 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" podUID="a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 29 23:10:32.670246 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:32.670198 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762"] Oct 29 23:10:32.670510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:32.670473 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" podUID="b384d2c7-ccaa-4655-a85a-2d1c16fc178a" containerName="maas-api" containerID="cri-o://5c2c2fb1712bbdbfc43dc5ff17ed2fae7fd1f786ecdb78e6fcfc8cd2786192af" gracePeriod=30 Oct 29 23:10:33.039679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.039644 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 29 23:10:33.039952 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.039917 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" podUID="bff30ab0-9c2d-41f0-902f-685ae8c67c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 29 23:10:33.096356 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.096321 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 29 23:10:33.096631 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.096604 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" podUID="b5d56fdf-4c0d-4243-980d-ae3e958544c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 29 23:10:33.153258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.148499 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 29 23:10:33.153258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.148559 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" podUID="a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 29 23:10:33.154283 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.154244 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp"] Oct 29 23:10:33.204213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.155770 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9bb12844-8806-48b3-af64-ec85f9f77eec" containerName="payload-pre-processing" Oct 29 23:10:33.229268 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.204440 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb12844-8806-48b3-af64-ec85f9f77eec" containerName="payload-pre-processing" Oct 29 23:10:33.233209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.229774 5057 cpu_manager.go:414] "containerMap: removing container" podUID="272e697e-08f1-44a5-838b-742a0f570253" containerName="payload-processing" Oct 29 23:10:33.262214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.237223 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="272e697e-08f1-44a5-838b-742a0f570253" containerName="payload-processing" Oct 29 23:10:33.262751 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.262718 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb12844-8806-48b3-af64-ec85f9f77eec" containerName="payload-pre-processing" Oct 29 23:10:33.270208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.266259 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="272e697e-08f1-44a5-838b-742a0f570253" containerName="payload-processing" Oct 29 23:10:33.346303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.315931 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:33.464257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.439136 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp"] Oct 29 23:10:33.842224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.442454 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503f6577-e8f6-474e-8b57-98e30edaa785-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:33.845254 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.842526 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:33.845580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.845544 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:33.853212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.845758 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503f6577-e8f6-474e-8b57-98e30edaa785-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:33.853508 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.853477 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:33.853723 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.853703 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/503f6577-e8f6-474e-8b57-98e30edaa785-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:33.853879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.853860 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:33.854045 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.854027 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:33.857944 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.854394 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-627wt\" (UniqueName: \"kubernetes.io/projected/503f6577-e8f6-474e-8b57-98e30edaa785-kube-api-access-627wt\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:33.951506 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.951444 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:33.997969 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:33.997922 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 23:10:34.144270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.132429 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503f6577-e8f6-474e-8b57-98e30edaa785-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:34.148355 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.148322 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:34.152359 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.152330 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/503f6577-e8f6-474e-8b57-98e30edaa785-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:34.287565 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.287526 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:34.452476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.419073 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:34.452476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.419271 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-627wt\" (UniqueName: \"kubernetes.io/projected/503f6577-e8f6-474e-8b57-98e30edaa785-kube-api-access-627wt\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:34.452476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.419379 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503f6577-e8f6-474e-8b57-98e30edaa785-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:34.452476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.419447 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:34.452476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.419519 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:34.452476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.420072 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:34.452476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.292103 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:34.452476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.220878 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:34.452476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.336324 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503f6577-e8f6-474e-8b57-98e30edaa785-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:34.452476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.336590 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 29 23:10:34.452476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.420620 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" podUID="a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 29 23:10:34.452476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.336649 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 29 23:10:34.452476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.421116 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" podUID="b5d56fdf-4c0d-4243-980d-ae3e958544c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 29 23:10:34.452476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.439748 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:34.452476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.440794 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503f6577-e8f6-474e-8b57-98e30edaa785-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:34.486434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.486384 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:34.490784 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.490738 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 29 23:10:34.491023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.490993 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" podUID="bff30ab0-9c2d-41f0-902f-685ae8c67c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 29 23:10:34.492559 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.492528 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-627wt\" (UniqueName: \"kubernetes.io/projected/503f6577-e8f6-474e-8b57-98e30edaa785-kube-api-access-627wt\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:34.496419 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:10:34.496394 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:10:34.506360 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.502988 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/503f6577-e8f6-474e-8b57-98e30edaa785-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:34.512796 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.512768 5057 generic.go:356] "Generic (PLEG): container finished" podID="b384d2c7-ccaa-4655-a85a-2d1c16fc178a" containerID="5c2c2fb1712bbdbfc43dc5ff17ed2fae7fd1f786ecdb78e6fcfc8cd2786192af" exitCode=0 Oct 29 23:10:34.577209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.574808 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:34.585465 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:34.585165 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" event={"ID":"b384d2c7-ccaa-4655-a85a-2d1c16fc178a","Type":"ContainerDied","Data":"5c2c2fb1712bbdbfc43dc5ff17ed2fae7fd1f786ecdb78e6fcfc8cd2786192af"} Oct 29 23:10:35.006252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.002394 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" Oct 29 23:10:35.022252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.021632 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 29 23:10:35.022252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.021698 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" podUID="b5d56fdf-4c0d-4243-980d-ae3e958544c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 29 23:10:35.049274 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.025678 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 29 23:10:35.049274 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.025727 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" podUID="bff30ab0-9c2d-41f0-902f-685ae8c67c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 29 23:10:35.191694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.191283 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-openshift-service-ca\") pod \"b384d2c7-ccaa-4655-a85a-2d1c16fc178a\" (UID: \"b384d2c7-ccaa-4655-a85a-2d1c16fc178a\") " Oct 29 23:10:35.191694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.191372 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-maas-api-tls\") pod \"b384d2c7-ccaa-4655-a85a-2d1c16fc178a\" (UID: \"b384d2c7-ccaa-4655-a85a-2d1c16fc178a\") " Oct 29 23:10:35.191694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.191441 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-kube-api-access-rcjjs\" (UniqueName: \"kubernetes.io/projected/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-kube-api-access-rcjjs\") pod \"b384d2c7-ccaa-4655-a85a-2d1c16fc178a\" (UID: \"b384d2c7-ccaa-4655-a85a-2d1c16fc178a\") " Oct 29 23:10:35.191694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.191639 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-metrics-certs\") pod \"b384d2c7-ccaa-4655-a85a-2d1c16fc178a\" (UID: \"b384d2c7-ccaa-4655-a85a-2d1c16fc178a\") " Oct 29 23:10:35.211876 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.192795 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 29 23:10:35.211876 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.192845 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" podUID="a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 29 23:10:35.211876 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.203399 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-openshift-service-ca" pod "b384d2c7-ccaa-4655-a85a-2d1c16fc178a" (UID: "b384d2c7-ccaa-4655-a85a-2d1c16fc178a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:35.289458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.287737 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-metrics-certs" pod "b384d2c7-ccaa-4655-a85a-2d1c16fc178a" (UID: "b384d2c7-ccaa-4655-a85a-2d1c16fc178a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:10:35.291740 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.291704 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-kube-api-access-rcjjs" pod "b384d2c7-ccaa-4655-a85a-2d1c16fc178a" (UID: "b384d2c7-ccaa-4655-a85a-2d1c16fc178a"). InnerVolumeSpecName "kube-api-access-rcjjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:35.292013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.291974 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-maas-api-tls" pod "b384d2c7-ccaa-4655-a85a-2d1c16fc178a" (UID: "b384d2c7-ccaa-4655-a85a-2d1c16fc178a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:10:35.295034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.294625 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:35.295034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.294651 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:35.295034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.294667 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:35.295034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.294684 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcjjs\" (UniqueName: \"kubernetes.io/projected/b384d2c7-ccaa-4655-a85a-2d1c16fc178a-kube-api-access-rcjjs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:35.483065 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:10:35.483012 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503f6577_e8f6_474e_8b57_98e30edaa785.slice/crio-60e83972324790e6433edddb6530c63ee86988c54ddb053a64c6c86b41ada010 WatchSource:0}: Error finding container 60e83972324790e6433edddb6530c63ee86988c54ddb053a64c6c86b41ada010: Status 404 returned error can't find the container with id 60e83972324790e6433edddb6530c63ee86988c54ddb053a64c6c86b41ada010 Oct 29 23:10:35.506549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.500222 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp"] Oct 29 23:10:35.507681 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.507580 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:35.508000 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.507940 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:35.508207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.508165 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:35.713786 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.709804 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" event={"ID":"503f6577-e8f6-474e-8b57-98e30edaa785","Type":"ContainerStarted","Data":"60e83972324790e6433edddb6530c63ee86988c54ddb053a64c6c86b41ada010"} Oct 29 23:10:35.713786 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.712382 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" event={"ID":"b384d2c7-ccaa-4655-a85a-2d1c16fc178a","Type":"ContainerDied","Data":"0097279e8fff87415496dad249cbd2a576fa345bda0a086af684717f5d83b72c"} Oct 29 23:10:35.713786 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.712450 5057 scope.go:122] "RemoveContainer" containerID="5c2c2fb1712bbdbfc43dc5ff17ed2fae7fd1f786ecdb78e6fcfc8cd2786192af" Oct 29 23:10:35.713786 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.712490 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762" Oct 29 23:10:35.911715 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.911665 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762"] Oct 29 23:10:35.961276 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:35.951881 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d21fd-6f68c975c-rx762"] Oct 29 23:10:36.015300 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:36.015249 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 29 23:10:36.015632 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:36.015576 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" podUID="bff30ab0-9c2d-41f0-902f-685ae8c67c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 29 23:10:36.228257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:36.227959 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 29 23:10:36.228257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:36.228020 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" podUID="a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 29 23:10:36.259219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:36.259113 5057 scope.go:122] "RemoveContainer" containerID="1dda5a8541a8a3d9c471f86ab24f64041d10795edfde8582cbb1943188ee4347" Oct 29 23:10:36.332342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:36.332314 5057 scope.go:122] "RemoveContainer" containerID="c5298ae8cb8b7fc1caf9bd2578f0796b8386381dca8d8ac29b7ee1bb4cc17f2e" Oct 29 23:10:36.741326 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:36.740174 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" event={"ID":"503f6577-e8f6-474e-8b57-98e30edaa785","Type":"ContainerStarted","Data":"d9096ca87b012ca5371d44c49d9e3ba229133e1217ec96b2df51d0fcca9b52da"} Oct 29 23:10:36.806211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:36.805562 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" podStartSLOduration=3.8055392489999997 podStartE2EDuration="3.805539249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:36.802554047 +0000 UTC m=+1683.454052579" watchObservedRunningTime="2026-10-29 23:10:36.805539249 +0000 UTC m=+1683.457037780" Oct 29 23:10:36.916042 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:36.904784 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:36.916042 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:36.906552 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:36.920826 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:36.920793 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:10:37.017462 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:37.016963 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:37.017611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:37.017586 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:37.018585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:37.018565 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:10:37.116628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:37.116586 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 29 23:10:37.116803 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:37.116654 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" podUID="a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 29 23:10:37.578229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:37.575182 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b384d2c7-ccaa-4655-a85a-2d1c16fc178a" path="/var/lib/kubelet/pods/b384d2c7-ccaa-4655-a85a-2d1c16fc178a/volumes" Oct 29 23:10:37.578229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:37.577074 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:37.578229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:37.577277 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 29 23:10:37.578229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:37.577355 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" podUID="503f6577-e8f6-474e-8b57-98e30edaa785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 29 23:10:38.299635 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:38.299604 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:38.300037 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:38.300018 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:38.300901 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:38.300883 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:10:38.576042 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:38.575965 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 29 23:10:38.576042 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:38.576021 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" podUID="503f6577-e8f6-474e-8b57-98e30edaa785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 29 23:10:39.578822 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:39.578786 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 29 23:10:39.579450 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:39.579396 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" podUID="503f6577-e8f6-474e-8b57-98e30edaa785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 29 23:10:40.576071 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:40.576034 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 29 23:10:40.576295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:40.576095 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" podUID="503f6577-e8f6-474e-8b57-98e30edaa785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 29 23:10:41.013669 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.013631 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm"] Oct 29 23:10:41.014726 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.014701 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b384d2c7-ccaa-4655-a85a-2d1c16fc178a" containerName="maas-api" Oct 29 23:10:41.014726 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.014725 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b384d2c7-ccaa-4655-a85a-2d1c16fc178a" containerName="maas-api" Oct 29 23:10:41.014945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.014930 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b384d2c7-ccaa-4655-a85a-2d1c16fc178a" containerName="maas-api" Oct 29 23:10:41.020235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.020204 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.076561 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.075886 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.076561 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.075952 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.076561 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.075995 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.076561 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.076123 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.076561 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.076213 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.076561 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.076267 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.076874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.076571 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.076874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.076618 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhrgg\" (UniqueName: \"kubernetes.io/projected/b68d1f22-af34-4928-9ea2-c89e5a881ff0-kube-api-access-fhrgg\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.076874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.076653 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.086035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.085996 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r"] Oct 29 23:10:41.095040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.095012 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.100365 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.100339 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm"] Oct 29 23:10:41.107462 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.107436 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r"] Oct 29 23:10:41.177991 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.177953 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.178169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.178018 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.178169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.178059 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.178169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.178088 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.178169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.178143 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a86092-f8c0-4608-8c15-9112358560f4-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.180382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.178281 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.180382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.178343 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a86092-f8c0-4608-8c15-9112358560f4-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.180382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.178371 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.180382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.178394 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a86092-f8c0-4608-8c15-9112358560f4-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.180382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.178442 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.180382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.178466 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8h5k\" (UniqueName: \"kubernetes.io/projected/71a86092-f8c0-4608-8c15-9112358560f4-kube-api-access-x8h5k\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.180382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.178628 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.180382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.178656 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.180382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.178717 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.180382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.178742 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhrgg\" (UniqueName: \"kubernetes.io/projected/b68d1f22-af34-4928-9ea2-c89e5a881ff0-kube-api-access-fhrgg\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.180382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.178768 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.180382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.178794 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.180382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.178848 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.180382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.179396 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.180382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.180098 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.181277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.180523 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.181277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.180605 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.181277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.180605 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.182671 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.182639 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.182896 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.182875 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.205474 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.205442 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.205630 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.205610 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhrgg\" (UniqueName: \"kubernetes.io/projected/b68d1f22-af34-4928-9ea2-c89e5a881ff0-kube-api-access-fhrgg\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.280358 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.280257 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8h5k\" (UniqueName: \"kubernetes.io/projected/71a86092-f8c0-4608-8c15-9112358560f4-kube-api-access-x8h5k\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.280516 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.280419 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.280516 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.280448 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.280661 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.280515 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.280661 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.280539 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.280661 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.280630 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.280825 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.280699 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a86092-f8c0-4608-8c15-9112358560f4-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.280886 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.280846 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.280940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.280850 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a86092-f8c0-4608-8c15-9112358560f4-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.280940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.280925 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a86092-f8c0-4608-8c15-9112358560f4-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.281043 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.280939 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.281043 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.280985 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.281150 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.281067 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.281585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.281563 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a86092-f8c0-4608-8c15-9112358560f4-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.282959 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.282939 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.283766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.283734 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a86092-f8c0-4608-8c15-9112358560f4-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.306711 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.306683 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a86092-f8c0-4608-8c15-9112358560f4-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.307037 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.307018 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8h5k\" (UniqueName: \"kubernetes.io/projected/71a86092-f8c0-4608-8c15-9112358560f4-kube-api-access-x8h5k\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.399801 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.399498 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:41.414002 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.411812 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:41.586139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.585642 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 29 23:10:41.586139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.585699 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" podUID="503f6577-e8f6-474e-8b57-98e30edaa785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 29 23:10:41.618066 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:41.618011 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm"] Oct 29 23:10:42.105277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:42.105233 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" podUID="7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" containerName="istio-proxy" containerID="cri-o://d2caf935fee120a25ff6fbc167cad2b963c5e6dd19edcd827f56bd131d21891b" gracePeriod=30 Oct 29 23:10:42.374867 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:42.374841 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r"] Oct 29 23:10:42.376310 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:10:42.376277 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a86092_f8c0_4608_8c15_9112358560f4.slice/crio-57e380d92330fb6733e491d00a6a0f95f3bc53d764625d90a178540ec247f8ec WatchSource:0}: Error finding container 57e380d92330fb6733e491d00a6a0f95f3bc53d764625d90a178540ec247f8ec: Status 404 returned error can't find the container with id 57e380d92330fb6733e491d00a6a0f95f3bc53d764625d90a178540ec247f8ec Oct 29 23:10:42.379447 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:42.379401 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:42.379546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:42.379489 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:42.379546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:42.379529 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:42.397177 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:10:42.397146 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68d1f22_af34_4928_9ea2_c89e5a881ff0.slice/crio-c2e9c7a6dea80d783b806af4b6929f5222845f3cf47b2781b03c3cf6e9078d74 WatchSource:0}: Error finding container c2e9c7a6dea80d783b806af4b6929f5222845f3cf47b2781b03c3cf6e9078d74: Status 404 returned error can't find the container with id c2e9c7a6dea80d783b806af4b6929f5222845f3cf47b2781b03c3cf6e9078d74 Oct 29 23:10:42.398693 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:42.398670 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm"] Oct 29 23:10:42.400470 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:42.400427 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:42.400573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:42.400507 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:42.400573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:42.400553 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:10:42.581357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:42.577383 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 29 23:10:42.581357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:42.577442 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" podUID="503f6577-e8f6-474e-8b57-98e30edaa785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 29 23:10:43.116252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:43.116209 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" event={"ID":"71a86092-f8c0-4608-8c15-9112358560f4","Type":"ContainerStarted","Data":"6d0abf1a5fbea40ad6d1d6cc419d9843cff6e04c56cba2c5e6184221f1498aca"} Oct 29 23:10:43.116689 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:43.116641 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" event={"ID":"71a86092-f8c0-4608-8c15-9112358560f4","Type":"ContainerStarted","Data":"57e380d92330fb6733e491d00a6a0f95f3bc53d764625d90a178540ec247f8ec"} Oct 29 23:10:43.119059 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:43.119026 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" event={"ID":"b68d1f22-af34-4928-9ea2-c89e5a881ff0","Type":"ContainerStarted","Data":"f538417a45d71fa19caf6747690d8161965a422cb4c18d8b2201843426a76ef4"} Oct 29 23:10:43.119238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:43.119081 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" event={"ID":"b68d1f22-af34-4928-9ea2-c89e5a881ff0","Type":"ContainerStarted","Data":"c2e9c7a6dea80d783b806af4b6929f5222845f3cf47b2781b03c3cf6e9078d74"} Oct 29 23:10:43.169521 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:43.169459 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" podStartSLOduration=2.169442272 podStartE2EDuration="2.169442272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:43.168149988 +0000 UTC m=+1689.819648521" watchObservedRunningTime="2026-10-29 23:10:43.169442272 +0000 UTC m=+1689.820940803" Oct 29 23:10:43.217574 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:43.217508 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" podStartSLOduration=3.217493953 podStartE2EDuration="3.217493953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:43.215628816 +0000 UTC m=+1689.867127348" watchObservedRunningTime="2026-10-29 23:10:43.217493953 +0000 UTC m=+1689.868992483" Oct 29 23:10:43.416269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:43.406357 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:43.420390 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:43.419965 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 29 23:10:43.420390 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:43.420034 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" podUID="b68d1f22-af34-4928-9ea2-c89e5a881ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 29 23:10:43.425714 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:43.425641 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:43.429785 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:43.429427 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:10:43.429785 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:43.429521 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" podUID="71a86092-f8c0-4608-8c15-9112358560f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 29 23:10:43.596210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:43.595175 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 29 23:10:43.596210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:43.595249 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" podUID="503f6577-e8f6-474e-8b57-98e30edaa785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 29 23:10:44.404066 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:44.404031 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 29 23:10:44.404602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:44.404079 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" podUID="b68d1f22-af34-4928-9ea2-c89e5a881ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 29 23:10:44.414228 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:44.414163 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:10:44.414389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:44.414246 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" podUID="71a86092-f8c0-4608-8c15-9112358560f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 29 23:10:44.690306 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:44.676520 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:44.690306 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:44.678439 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:44.690306 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:44.684411 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:10:45.418210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:45.418138 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:10:45.418702 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:45.418231 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" podUID="71a86092-f8c0-4608-8c15-9112358560f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 29 23:10:45.431225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:45.421565 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 29 23:10:45.431225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:45.421635 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" podUID="b68d1f22-af34-4928-9ea2-c89e5a881ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 29 23:10:46.401309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:46.401276 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 29 23:10:46.401519 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:46.401499 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" podUID="b68d1f22-af34-4928-9ea2-c89e5a881ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 29 23:10:46.412750 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:46.412722 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:10:46.412881 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:46.412780 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" podUID="71a86092-f8c0-4608-8c15-9112358560f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 29 23:10:46.972212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:46.971331 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:10:46.972212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:46.971393 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" podUID="7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:10:47.400807 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.400741 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 29 23:10:47.401064 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.401024 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" podUID="b68d1f22-af34-4928-9ea2-c89e5a881ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 29 23:10:47.412901 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.412869 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:10:47.413033 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.412945 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" podUID="71a86092-f8c0-4608-8c15-9112358560f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 29 23:10:47.909093 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.909060 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:10:47.992343 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.992303 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-workload-certs\") pod \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " Oct 29 23:10:47.992851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.992359 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istiod-ca-cert\") pod \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " Oct 29 23:10:47.992851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.992464 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-data\") pod \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " Oct 29 23:10:47.992851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.992497 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-podinfo\") pod \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " Oct 29 23:10:47.992851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.992664 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-token\" (UniqueName: \"kubernetes.io/projected/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-token\") pod \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " Oct 29 23:10:47.992851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.992738 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-workload-socket\") pod \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " Oct 29 23:10:47.992851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.992800 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-envoy\") pod \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " Oct 29 23:10:47.992851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.992837 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-kube-api-access-cmfgk\" (UniqueName: \"kubernetes.io/projected/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-kube-api-access-cmfgk\") pod \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " Oct 29 23:10:47.993249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.992877 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-credential-socket\") pod \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\" (UID: \"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33\") " Oct 29 23:10:47.993818 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.993752 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-workload-certs" pod "7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" (UID: "7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:47.993818 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.993811 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-credential-socket" pod "7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" (UID: "7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:47.994008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.993976 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-workload-socket" pod "7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" (UID: "7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:47.994468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.994440 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-data" pod "7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" (UID: "7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:47.996834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.996805 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istiod-ca-cert" pod "7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" (UID: "7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:47.997264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.997238 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-podinfo" pod "7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" (UID: "7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:10:47.997365 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:47.997344 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-kube-api-access-cmfgk" pod "7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" (UID: "7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33"). InnerVolumeSpecName "kube-api-access-cmfgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:48.003409 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.003375 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-envoy" pod "7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" (UID: "7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:48.003535 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.003436 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-token" pod "7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" (UID: "7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:48.098264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.097635 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:48.098264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.097670 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:48.098264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.097685 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:48.098264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.097698 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmfgk\" (UniqueName: \"kubernetes.io/projected/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-kube-api-access-cmfgk\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:48.098264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.097715 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:48.098264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.097729 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:48.098264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.097742 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:48.098264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.097754 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:48.098264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.097766 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:48.400751 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.400063 5057 generic.go:356] "Generic (PLEG): container finished" podID="7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" containerID="d2caf935fee120a25ff6fbc167cad2b963c5e6dd19edcd827f56bd131d21891b" exitCode=0 Oct 29 23:10:48.400751 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.400129 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" Oct 29 23:10:48.400751 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.400265 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" event={"ID":"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33","Type":"ContainerDied","Data":"d2caf935fee120a25ff6fbc167cad2b963c5e6dd19edcd827f56bd131d21891b"} Oct 29 23:10:48.400751 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.400304 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm" event={"ID":"7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33","Type":"ContainerDied","Data":"86b195beae22aabac90259825fc9a21b9e909f30fafa51398a2764388729f0d0"} Oct 29 23:10:48.400751 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.400324 5057 scope.go:122] "RemoveContainer" containerID="d2caf935fee120a25ff6fbc167cad2b963c5e6dd19edcd827f56bd131d21891b" Oct 29 23:10:48.401709 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.401406 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 29 23:10:48.401709 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.401458 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" podUID="b68d1f22-af34-4928-9ea2-c89e5a881ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 29 23:10:48.413926 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.413091 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:10:48.413926 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.413154 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" podUID="71a86092-f8c0-4608-8c15-9112358560f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 29 23:10:48.417084 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.417039 5057 scope.go:122] "RemoveContainer" containerID="027e0c714b6b67a7b970559717a1af03d6908446225b0b4bf54126b5cf7df370" Oct 29 23:10:48.501610 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.501585 5057 scope.go:122] "RemoveContainer" containerID="d2caf935fee120a25ff6fbc167cad2b963c5e6dd19edcd827f56bd131d21891b" Oct 29 23:10:48.502096 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:10:48.502041 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2caf935fee120a25ff6fbc167cad2b963c5e6dd19edcd827f56bd131d21891b\": container with ID starting with d2caf935fee120a25ff6fbc167cad2b963c5e6dd19edcd827f56bd131d21891b not found: ID does not exist" containerID="d2caf935fee120a25ff6fbc167cad2b963c5e6dd19edcd827f56bd131d21891b" Oct 29 23:10:48.502096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.502083 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2caf935fee120a25ff6fbc167cad2b963c5e6dd19edcd827f56bd131d21891b"} err="failed to get container status \"d2caf935fee120a25ff6fbc167cad2b963c5e6dd19edcd827f56bd131d21891b\": rpc error: code = NotFound desc = could not find container \"d2caf935fee120a25ff6fbc167cad2b963c5e6dd19edcd827f56bd131d21891b\": container with ID starting with d2caf935fee120a25ff6fbc167cad2b963c5e6dd19edcd827f56bd131d21891b not found: ID does not exist" Oct 29 23:10:48.502371 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.502109 5057 scope.go:122] "RemoveContainer" containerID="027e0c714b6b67a7b970559717a1af03d6908446225b0b4bf54126b5cf7df370" Oct 29 23:10:48.502517 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:10:48.502481 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027e0c714b6b67a7b970559717a1af03d6908446225b0b4bf54126b5cf7df370\": container with ID starting with 027e0c714b6b67a7b970559717a1af03d6908446225b0b4bf54126b5cf7df370 not found: ID does not exist" containerID="027e0c714b6b67a7b970559717a1af03d6908446225b0b4bf54126b5cf7df370" Oct 29 23:10:48.502608 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.502513 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027e0c714b6b67a7b970559717a1af03d6908446225b0b4bf54126b5cf7df370"} err="failed to get container status \"027e0c714b6b67a7b970559717a1af03d6908446225b0b4bf54126b5cf7df370\": rpc error: code = NotFound desc = could not find container \"027e0c714b6b67a7b970559717a1af03d6908446225b0b4bf54126b5cf7df370\": container with ID starting with 027e0c714b6b67a7b970559717a1af03d6908446225b0b4bf54126b5cf7df370 not found: ID does not exist" Oct 29 23:10:48.503025 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.503011 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm"] Oct 29 23:10:48.506935 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:48.506912 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-6d21fd-openshift-default-8c6b4b7f7-z9frm"] Oct 29 23:10:49.400815 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:49.400775 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 29 23:10:49.401314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:49.400844 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" podUID="b68d1f22-af34-4928-9ea2-c89e5a881ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 29 23:10:49.412948 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:49.412913 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:10:49.413107 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:49.412984 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" podUID="71a86092-f8c0-4608-8c15-9112358560f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 29 23:10:49.483420 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:49.483390 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" path="/var/lib/kubelet/pods/7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33/volumes" Oct 29 23:10:49.968926 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:49.968888 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59"] Oct 29 23:10:49.970100 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:49.970066 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59" podUID="884eed63-5693-41b6-81c6-0d4cd1abdc4c" containerName="payload-processing" containerID="cri-o://3d9145caff7e04764bf000d51ca9e9edc181b8d03e584c78bdf48135ab4cc0d6" gracePeriod=30 Oct 29 23:10:49.988867 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:49.988838 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8"] Oct 29 23:10:49.989421 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:49.989065 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8" podUID="a6e31a55-feaa-4b94-bb1f-5f7758706537" containerName="payload-pre-processing" containerID="cri-o://261570d035c22aad0faeadebd21a01b15bef2728d4dd09647c621b1709263776" gracePeriod=30 Oct 29 23:10:50.528510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:50.504864 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:10:50.528510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:50.504918 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" podUID="71a86092-f8c0-4608-8c15-9112358560f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 29 23:10:50.640836 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:50.640745 5057 generic.go:356] "Generic (PLEG): container finished" podID="a6e31a55-feaa-4b94-bb1f-5f7758706537" containerID="261570d035c22aad0faeadebd21a01b15bef2728d4dd09647c621b1709263776" exitCode=0 Oct 29 23:10:50.641252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:50.641222 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8" event={"ID":"a6e31a55-feaa-4b94-bb1f-5f7758706537","Type":"ContainerDied","Data":"261570d035c22aad0faeadebd21a01b15bef2728d4dd09647c621b1709263776"} Oct 29 23:10:50.644943 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:50.644918 5057 generic.go:356] "Generic (PLEG): container finished" podID="884eed63-5693-41b6-81c6-0d4cd1abdc4c" containerID="3d9145caff7e04764bf000d51ca9e9edc181b8d03e584c78bdf48135ab4cc0d6" exitCode=0 Oct 29 23:10:50.645254 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:50.645233 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59" event={"ID":"884eed63-5693-41b6-81c6-0d4cd1abdc4c","Type":"ContainerDied","Data":"3d9145caff7e04764bf000d51ca9e9edc181b8d03e584c78bdf48135ab4cc0d6"} Oct 29 23:10:50.805092 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:50.804499 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8" Oct 29 23:10:50.808326 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:50.808299 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59" Oct 29 23:10:50.871947 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:50.871902 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e31a55-feaa-4b94-bb1f-5f7758706537-kube-api-access-l9w9b\" (UniqueName: \"kubernetes.io/projected/a6e31a55-feaa-4b94-bb1f-5f7758706537-kube-api-access-l9w9b\") pod \"a6e31a55-feaa-4b94-bb1f-5f7758706537\" (UID: \"a6e31a55-feaa-4b94-bb1f-5f7758706537\") " Oct 29 23:10:50.874903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:50.872463 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884eed63-5693-41b6-81c6-0d4cd1abdc4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884eed63-5693-41b6-81c6-0d4cd1abdc4c-plugins-config-volume\") pod \"884eed63-5693-41b6-81c6-0d4cd1abdc4c\" (UID: \"884eed63-5693-41b6-81c6-0d4cd1abdc4c\") " Oct 29 23:10:50.878127 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:50.875116 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884eed63-5693-41b6-81c6-0d4cd1abdc4c-kube-api-access-v72kr\" (UniqueName: \"kubernetes.io/projected/884eed63-5693-41b6-81c6-0d4cd1abdc4c-kube-api-access-v72kr\") pod \"884eed63-5693-41b6-81c6-0d4cd1abdc4c\" (UID: \"884eed63-5693-41b6-81c6-0d4cd1abdc4c\") " Oct 29 23:10:50.878127 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:50.876394 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e31a55-feaa-4b94-bb1f-5f7758706537-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e31a55-feaa-4b94-bb1f-5f7758706537-plugins-config-volume\") pod \"a6e31a55-feaa-4b94-bb1f-5f7758706537\" (UID: \"a6e31a55-feaa-4b94-bb1f-5f7758706537\") " Oct 29 23:10:50.878127 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:50.877802 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e31a55-feaa-4b94-bb1f-5f7758706537-plugins-config-volume" pod "a6e31a55-feaa-4b94-bb1f-5f7758706537" (UID: "a6e31a55-feaa-4b94-bb1f-5f7758706537"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:50.879337 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:50.878418 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884eed63-5693-41b6-81c6-0d4cd1abdc4c-plugins-config-volume" pod "884eed63-5693-41b6-81c6-0d4cd1abdc4c" (UID: "884eed63-5693-41b6-81c6-0d4cd1abdc4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:50.886203 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:50.884503 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e31a55-feaa-4b94-bb1f-5f7758706537-kube-api-access-l9w9b" pod "a6e31a55-feaa-4b94-bb1f-5f7758706537" (UID: "a6e31a55-feaa-4b94-bb1f-5f7758706537"). InnerVolumeSpecName "kube-api-access-l9w9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:50.886203 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:50.884958 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884eed63-5693-41b6-81c6-0d4cd1abdc4c-kube-api-access-v72kr" pod "884eed63-5693-41b6-81c6-0d4cd1abdc4c" (UID: "884eed63-5693-41b6-81c6-0d4cd1abdc4c"). InnerVolumeSpecName "kube-api-access-v72kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:50.998630 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:50.998588 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9w9b\" (UniqueName: \"kubernetes.io/projected/a6e31a55-feaa-4b94-bb1f-5f7758706537-kube-api-access-l9w9b\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:50.998801 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:50.998788 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884eed63-5693-41b6-81c6-0d4cd1abdc4c-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:50.998871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:50.998863 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v72kr\" (UniqueName: \"kubernetes.io/projected/884eed63-5693-41b6-81c6-0d4cd1abdc4c-kube-api-access-v72kr\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:50.998927 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:50.998919 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e31a55-feaa-4b94-bb1f-5f7758706537-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:10:51.484997 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:51.484958 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:10:51.485181 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:51.485018 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" podUID="b68d1f22-af34-4928-9ea2-c89e5a881ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:51.519468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:51.519435 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:51.519953 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:51.519928 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:51.521023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:51.521003 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:10:51.664079 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:51.664048 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:51.665663 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:51.665640 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:51.670573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:51.667493 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:10:51.670573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:51.669441 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59" event={"ID":"884eed63-5693-41b6-81c6-0d4cd1abdc4c","Type":"ContainerDied","Data":"f15e0a3f10866757acec372cb559f5700b080b207779ded47be9d0b53d67ca53"} Oct 29 23:10:51.670573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:51.669495 5057 scope.go:122] "RemoveContainer" containerID="3d9145caff7e04764bf000d51ca9e9edc181b8d03e584c78bdf48135ab4cc0d6" Oct 29 23:10:51.670573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:51.669639 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59" Oct 29 23:10:51.677881 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:51.677410 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8" Oct 29 23:10:51.677881 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:51.677498 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8" event={"ID":"a6e31a55-feaa-4b94-bb1f-5f7758706537","Type":"ContainerDied","Data":"88864bda60277be3daa19df843c7f624badf5bdb40f81853a4bb395eb25e30a3"} Oct 29 23:10:51.722807 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:51.722750 5057 scope.go:122] "RemoveContainer" containerID="261570d035c22aad0faeadebd21a01b15bef2728d4dd09647c621b1709263776" Oct 29 23:10:51.776949 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:51.776892 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59"] Oct 29 23:10:51.780692 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:51.780668 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-921680-cc9b96bdb-gjp59"] Oct 29 23:10:51.813037 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:51.813007 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8"] Oct 29 23:10:51.817417 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:51.817390 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-785d8cd8f5-ddhc8"] Oct 29 23:10:53.482390 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:53.482359 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884eed63-5693-41b6-81c6-0d4cd1abdc4c" path="/var/lib/kubelet/pods/884eed63-5693-41b6-81c6-0d4cd1abdc4c/volumes" Oct 29 23:10:53.482903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:53.482888 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e31a55-feaa-4b94-bb1f-5f7758706537" path="/var/lib/kubelet/pods/a6e31a55-feaa-4b94-bb1f-5f7758706537/volumes" Oct 29 23:10:56.248077 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:10:56.248039 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:11:03.917393 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:11:03.917347 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:11:10.555754 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:10.555718 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm"] Oct 29 23:11:10.556277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:10.555947 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" podUID="5409eeb6-ca88-4c97-9ee9-16449a4f9dd6" containerName="maas-api" containerID="cri-o://11a755063c20690ca3f54eb044f81739975eb9f02702d670a26f801080425697" gracePeriod=30 Oct 29 23:11:11.334535 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:11.334500 5057 generic.go:356] "Generic (PLEG): container finished" podID="5409eeb6-ca88-4c97-9ee9-16449a4f9dd6" containerID="11a755063c20690ca3f54eb044f81739975eb9f02702d670a26f801080425697" exitCode=0 Oct 29 23:11:11.334998 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:11.334974 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" event={"ID":"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6","Type":"ContainerDied","Data":"11a755063c20690ca3f54eb044f81739975eb9f02702d670a26f801080425697"} Oct 29 23:11:11.340617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:11.340248 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" Oct 29 23:11:11.419714 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:11.419622 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-kube-api-access-jmrd6\" (UniqueName: \"kubernetes.io/projected/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-kube-api-access-jmrd6\") pod \"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6\" (UID: \"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6\") " Oct 29 23:11:11.419890 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:11.419727 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-maas-api-tls\") pod \"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6\" (UID: \"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6\") " Oct 29 23:11:11.419890 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:11.419771 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-metrics-certs\") pod \"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6\" (UID: \"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6\") " Oct 29 23:11:11.420010 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:11.419952 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-openshift-service-ca\") pod \"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6\" (UID: \"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6\") " Oct 29 23:11:11.420412 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:11.420380 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-openshift-service-ca" pod "5409eeb6-ca88-4c97-9ee9-16449a4f9dd6" (UID: "5409eeb6-ca88-4c97-9ee9-16449a4f9dd6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:11.420583 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:11.420563 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:11.422101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:11.422081 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-metrics-certs" pod "5409eeb6-ca88-4c97-9ee9-16449a4f9dd6" (UID: "5409eeb6-ca88-4c97-9ee9-16449a4f9dd6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:11:11.422426 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:11.422400 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-kube-api-access-jmrd6" pod "5409eeb6-ca88-4c97-9ee9-16449a4f9dd6" (UID: "5409eeb6-ca88-4c97-9ee9-16449a4f9dd6"). InnerVolumeSpecName "kube-api-access-jmrd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:11.422698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:11.422673 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-maas-api-tls" pod "5409eeb6-ca88-4c97-9ee9-16449a4f9dd6" (UID: "5409eeb6-ca88-4c97-9ee9-16449a4f9dd6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:11:11.524202 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:11.521648 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmrd6\" (UniqueName: \"kubernetes.io/projected/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-kube-api-access-jmrd6\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:11.524202 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:11.521679 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:11.524202 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:11.521692 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:12.362844 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:12.362807 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" event={"ID":"5409eeb6-ca88-4c97-9ee9-16449a4f9dd6","Type":"ContainerDied","Data":"b19c47cbcd30c00ecbb1ce82b3a9bdfb3974a092e1693156d31c240658cef67d"} Oct 29 23:11:12.363491 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:12.363375 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm" Oct 29 23:11:12.363592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:12.363469 5057 scope.go:122] "RemoveContainer" containerID="11a755063c20690ca3f54eb044f81739975eb9f02702d670a26f801080425697" Oct 29 23:11:12.423436 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:12.423406 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm"] Oct 29 23:11:12.427526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:12.427496 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-85c702-69d857cf64-8phlm"] Oct 29 23:11:13.258405 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:13.258370 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v"] Oct 29 23:11:13.258689 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:13.258640 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v" podUID="cdf4f520-c1fb-4178-9685-2ddf12262613" containerName="payload-processing" containerID="cri-o://ad6254ddc3c11b7ed79a940383ddcca5fc3594f5b44a9e79bd2695f8cbffa7d5" gracePeriod=30 Oct 29 23:11:13.290635 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:13.290593 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv"] Oct 29 23:11:13.290915 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:13.290882 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv" podUID="e7897e3b-8c0e-45be-a3b1-ad5ccf299283" containerName="payload-pre-processing" containerID="cri-o://2ce97c606bddb4fd86fd6f99b161c90cbc5bdd8c34e6293e4c2530bbc2a1c8c0" gracePeriod=30 Oct 29 23:11:13.488588 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:13.488561 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5409eeb6-ca88-4c97-9ee9-16449a4f9dd6" path="/var/lib/kubelet/pods/5409eeb6-ca88-4c97-9ee9-16449a4f9dd6/volumes" Oct 29 23:11:13.932433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:13.930953 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v" Oct 29 23:11:13.991586 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:13.991562 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv" Oct 29 23:11:14.023322 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.023287 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf4f520-c1fb-4178-9685-2ddf12262613-kube-api-access-ng5q9\" (UniqueName: \"kubernetes.io/projected/cdf4f520-c1fb-4178-9685-2ddf12262613-kube-api-access-ng5q9\") pod \"cdf4f520-c1fb-4178-9685-2ddf12262613\" (UID: \"cdf4f520-c1fb-4178-9685-2ddf12262613\") " Oct 29 23:11:14.023534 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.023372 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7897e3b-8c0e-45be-a3b1-ad5ccf299283-kube-api-access-sthjc\" (UniqueName: \"kubernetes.io/projected/e7897e3b-8c0e-45be-a3b1-ad5ccf299283-kube-api-access-sthjc\") pod \"e7897e3b-8c0e-45be-a3b1-ad5ccf299283\" (UID: \"e7897e3b-8c0e-45be-a3b1-ad5ccf299283\") " Oct 29 23:11:14.023806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.023782 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdf4f520-c1fb-4178-9685-2ddf12262613-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf4f520-c1fb-4178-9685-2ddf12262613-plugins-config-volume\") pod \"cdf4f520-c1fb-4178-9685-2ddf12262613\" (UID: \"cdf4f520-c1fb-4178-9685-2ddf12262613\") " Oct 29 23:11:14.023972 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.023946 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7897e3b-8c0e-45be-a3b1-ad5ccf299283-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7897e3b-8c0e-45be-a3b1-ad5ccf299283-plugins-config-volume\") pod \"e7897e3b-8c0e-45be-a3b1-ad5ccf299283\" (UID: \"e7897e3b-8c0e-45be-a3b1-ad5ccf299283\") " Oct 29 23:11:14.024442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.024415 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf4f520-c1fb-4178-9685-2ddf12262613-plugins-config-volume" pod "cdf4f520-c1fb-4178-9685-2ddf12262613" (UID: "cdf4f520-c1fb-4178-9685-2ddf12262613"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:14.024709 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.024679 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7897e3b-8c0e-45be-a3b1-ad5ccf299283-plugins-config-volume" pod "e7897e3b-8c0e-45be-a3b1-ad5ccf299283" (UID: "e7897e3b-8c0e-45be-a3b1-ad5ccf299283"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:14.025334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.025307 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7897e3b-8c0e-45be-a3b1-ad5ccf299283-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:14.025435 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.025337 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf4f520-c1fb-4178-9685-2ddf12262613-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:14.026223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.026161 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7897e3b-8c0e-45be-a3b1-ad5ccf299283-kube-api-access-sthjc" pod "e7897e3b-8c0e-45be-a3b1-ad5ccf299283" (UID: "e7897e3b-8c0e-45be-a3b1-ad5ccf299283"). InnerVolumeSpecName "kube-api-access-sthjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:14.026308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.026239 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf4f520-c1fb-4178-9685-2ddf12262613-kube-api-access-ng5q9" pod "cdf4f520-c1fb-4178-9685-2ddf12262613" (UID: "cdf4f520-c1fb-4178-9685-2ddf12262613"). InnerVolumeSpecName "kube-api-access-ng5q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:14.127305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.127270 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng5q9\" (UniqueName: \"kubernetes.io/projected/cdf4f520-c1fb-4178-9685-2ddf12262613-kube-api-access-ng5q9\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:14.127305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.127305 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sthjc\" (UniqueName: \"kubernetes.io/projected/e7897e3b-8c0e-45be-a3b1-ad5ccf299283-kube-api-access-sthjc\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:14.397415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.390714 5057 generic.go:356] "Generic (PLEG): container finished" podID="e7897e3b-8c0e-45be-a3b1-ad5ccf299283" containerID="2ce97c606bddb4fd86fd6f99b161c90cbc5bdd8c34e6293e4c2530bbc2a1c8c0" exitCode=0 Oct 29 23:11:14.397415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.390893 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv" event={"ID":"e7897e3b-8c0e-45be-a3b1-ad5ccf299283","Type":"ContainerDied","Data":"2ce97c606bddb4fd86fd6f99b161c90cbc5bdd8c34e6293e4c2530bbc2a1c8c0"} Oct 29 23:11:14.397415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.390950 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv" event={"ID":"e7897e3b-8c0e-45be-a3b1-ad5ccf299283","Type":"ContainerDied","Data":"4b03feb3eea734abdaf607db81871e75d4537649dbafce70d728f9cf7965b694"} Oct 29 23:11:14.397415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.391074 5057 scope.go:122] "RemoveContainer" containerID="2ce97c606bddb4fd86fd6f99b161c90cbc5bdd8c34e6293e4c2530bbc2a1c8c0" Oct 29 23:11:14.397415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.391226 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv" Oct 29 23:11:14.397415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.395741 5057 generic.go:356] "Generic (PLEG): container finished" podID="cdf4f520-c1fb-4178-9685-2ddf12262613" containerID="ad6254ddc3c11b7ed79a940383ddcca5fc3594f5b44a9e79bd2695f8cbffa7d5" exitCode=0 Oct 29 23:11:14.397415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.395792 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v" event={"ID":"cdf4f520-c1fb-4178-9685-2ddf12262613","Type":"ContainerDied","Data":"ad6254ddc3c11b7ed79a940383ddcca5fc3594f5b44a9e79bd2695f8cbffa7d5"} Oct 29 23:11:14.397415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.395836 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v" event={"ID":"cdf4f520-c1fb-4178-9685-2ddf12262613","Type":"ContainerDied","Data":"6e6194d2a2cef3315fccb004c55269e1b02f121d31172278f7a34365e11f931e"} Oct 29 23:11:14.397415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.395911 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v" Oct 29 23:11:14.440447 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.439893 5057 scope.go:122] "RemoveContainer" containerID="2ce97c606bddb4fd86fd6f99b161c90cbc5bdd8c34e6293e4c2530bbc2a1c8c0" Oct 29 23:11:14.440996 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:11:14.440636 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ce97c606bddb4fd86fd6f99b161c90cbc5bdd8c34e6293e4c2530bbc2a1c8c0\": container with ID starting with 2ce97c606bddb4fd86fd6f99b161c90cbc5bdd8c34e6293e4c2530bbc2a1c8c0 not found: ID does not exist" containerID="2ce97c606bddb4fd86fd6f99b161c90cbc5bdd8c34e6293e4c2530bbc2a1c8c0" Oct 29 23:11:14.440996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.440674 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ce97c606bddb4fd86fd6f99b161c90cbc5bdd8c34e6293e4c2530bbc2a1c8c0"} err="failed to get container status \"2ce97c606bddb4fd86fd6f99b161c90cbc5bdd8c34e6293e4c2530bbc2a1c8c0\": rpc error: code = NotFound desc = could not find container \"2ce97c606bddb4fd86fd6f99b161c90cbc5bdd8c34e6293e4c2530bbc2a1c8c0\": container with ID starting with 2ce97c606bddb4fd86fd6f99b161c90cbc5bdd8c34e6293e4c2530bbc2a1c8c0 not found: ID does not exist" Oct 29 23:11:14.440996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.440697 5057 scope.go:122] "RemoveContainer" containerID="ad6254ddc3c11b7ed79a940383ddcca5fc3594f5b44a9e79bd2695f8cbffa7d5" Oct 29 23:11:14.486566 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.486517 5057 scope.go:122] "RemoveContainer" containerID="ad6254ddc3c11b7ed79a940383ddcca5fc3594f5b44a9e79bd2695f8cbffa7d5" Oct 29 23:11:14.487119 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:11:14.487068 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad6254ddc3c11b7ed79a940383ddcca5fc3594f5b44a9e79bd2695f8cbffa7d5\": container with ID starting with ad6254ddc3c11b7ed79a940383ddcca5fc3594f5b44a9e79bd2695f8cbffa7d5 not found: ID does not exist" containerID="ad6254ddc3c11b7ed79a940383ddcca5fc3594f5b44a9e79bd2695f8cbffa7d5" Oct 29 23:11:14.487119 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.487103 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad6254ddc3c11b7ed79a940383ddcca5fc3594f5b44a9e79bd2695f8cbffa7d5"} err="failed to get container status \"ad6254ddc3c11b7ed79a940383ddcca5fc3594f5b44a9e79bd2695f8cbffa7d5\": rpc error: code = NotFound desc = could not find container \"ad6254ddc3c11b7ed79a940383ddcca5fc3594f5b44a9e79bd2695f8cbffa7d5\": container with ID starting with ad6254ddc3c11b7ed79a940383ddcca5fc3594f5b44a9e79bd2695f8cbffa7d5 not found: ID does not exist" Oct 29 23:11:14.488953 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.488930 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv"] Oct 29 23:11:14.506351 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.495154 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-5bd488b8bc-ttnmv"] Oct 29 23:11:14.551308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.543038 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v"] Oct 29 23:11:14.551987 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:14.551963 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-669dcc7b56-k5p2v"] Oct 29 23:11:15.482326 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:15.482293 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdf4f520-c1fb-4178-9685-2ddf12262613" path="/var/lib/kubelet/pods/cdf4f520-c1fb-4178-9685-2ddf12262613/volumes" Oct 29 23:11:15.482764 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:15.482749 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7897e3b-8c0e-45be-a3b1-ad5ccf299283" path="/var/lib/kubelet/pods/e7897e3b-8c0e-45be-a3b1-ad5ccf299283/volumes" Oct 29 23:11:19.047369 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:19.047330 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl"] Oct 29 23:11:19.048301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:19.048239 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" podUID="72b4ea44-8863-40e5-baee-a5e1e0c94285" containerName="istio-proxy" containerID="cri-o://ae3e383566ea7c6451ef12660a6ae7e23e8e30e2ddb69b02bfe85dd52471b02b" gracePeriod=30 Oct 29 23:11:19.768237 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:19.763284 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:19.926244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:19.919776 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 23:11:20.413335 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:20.413288 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:20.466302 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:20.466255 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:20.526332 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:20.526286 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 23:11:20.765473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:20.765423 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:20.766410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:20.766386 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:11:21.036308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:21.029137 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:21.036308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:21.030340 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:11:21.441740 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:21.439317 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:11:21.441740 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:21.439976 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:11:21.566894 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:21.566835 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:21.789267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:21.782292 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:22.145267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:22.132698 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:22.332623 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:22.332398 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:11:22.563673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:22.526286 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:22.633213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:22.628337 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:11:22.633213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:22.628403 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" podUID="b68d1f22-af34-4928-9ea2-c89e5a881ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:22.666213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:22.660458 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:22.781212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:22.780316 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:11:22.781212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:22.780413 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" podUID="71a86092-f8c0-4608-8c15-9112358560f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:22.804693 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:22.792328 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:23.218208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:23.147109 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:23.246950 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:23.246896 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:11:23.247287 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:23.247237 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" podUID="bff30ab0-9c2d-41f0-902f-685ae8c67c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:23.247842 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:23.247818 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:11:23.248022 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:23.247998 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" podUID="b5d56fdf-4c0d-4243-980d-ae3e958544c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:23.616273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:23.603325 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:11:23.667745 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:23.667699 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:23.972219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:23.939408 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:24.187352 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:24.187304 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 23:11:24.368270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:24.365597 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:11:24.368270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:24.365661 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" podUID="a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:24.598211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:24.595447 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:24.650636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:24.647444 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:11:24.705629 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:24.705573 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:24.897116 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:24.897085 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:11:24.943874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:24.943842 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:11:25.044529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.044496 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:11:25.093403 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.080778 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:11:25.360701 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.360670 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:11:25.509546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.509351 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-envoy\") pod \"72b4ea44-8863-40e5-baee-a5e1e0c94285\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " Oct 29 23:11:25.509828 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.509794 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-workload-certs\") pod \"72b4ea44-8863-40e5-baee-a5e1e0c94285\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " Oct 29 23:11:25.509927 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.509873 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-podinfo\") pod \"72b4ea44-8863-40e5-baee-a5e1e0c94285\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " Oct 29 23:11:25.509927 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.509916 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-data\") pod \"72b4ea44-8863-40e5-baee-a5e1e0c94285\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " Oct 29 23:11:25.510009 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.509995 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-token\" (UniqueName: \"kubernetes.io/projected/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-token\") pod \"72b4ea44-8863-40e5-baee-a5e1e0c94285\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " Oct 29 23:11:25.510080 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.510063 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-workload-socket\") pod \"72b4ea44-8863-40e5-baee-a5e1e0c94285\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " Oct 29 23:11:25.510389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.510365 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b4ea44-8863-40e5-baee-a5e1e0c94285-kube-api-access-x7lr9\" (UniqueName: \"kubernetes.io/projected/72b4ea44-8863-40e5-baee-a5e1e0c94285-kube-api-access-x7lr9\") pod \"72b4ea44-8863-40e5-baee-a5e1e0c94285\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " Oct 29 23:11:25.510549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.510435 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-credential-socket\") pod \"72b4ea44-8863-40e5-baee-a5e1e0c94285\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " Oct 29 23:11:25.510638 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.510574 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72b4ea44-8863-40e5-baee-a5e1e0c94285-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b4ea44-8863-40e5-baee-a5e1e0c94285-istiod-ca-cert\") pod \"72b4ea44-8863-40e5-baee-a5e1e0c94285\" (UID: \"72b4ea44-8863-40e5-baee-a5e1e0c94285\") " Oct 29 23:11:25.512947 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.512920 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-data" pod "72b4ea44-8863-40e5-baee-a5e1e0c94285" (UID: "72b4ea44-8863-40e5-baee-a5e1e0c94285"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:25.514523 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.514491 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b4ea44-8863-40e5-baee-a5e1e0c94285-istiod-ca-cert" pod "72b4ea44-8863-40e5-baee-a5e1e0c94285" (UID: "72b4ea44-8863-40e5-baee-a5e1e0c94285"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:25.517133 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.517109 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-workload-socket" pod "72b4ea44-8863-40e5-baee-a5e1e0c94285" (UID: "72b4ea44-8863-40e5-baee-a5e1e0c94285"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:25.523851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.523417 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-credential-socket" pod "72b4ea44-8863-40e5-baee-a5e1e0c94285" (UID: "72b4ea44-8863-40e5-baee-a5e1e0c94285"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:25.523851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.523549 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-token" pod "72b4ea44-8863-40e5-baee-a5e1e0c94285" (UID: "72b4ea44-8863-40e5-baee-a5e1e0c94285"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:25.551343 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.551298 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-workload-certs" pod "72b4ea44-8863-40e5-baee-a5e1e0c94285" (UID: "72b4ea44-8863-40e5-baee-a5e1e0c94285"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:25.555376 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.554414 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b4ea44-8863-40e5-baee-a5e1e0c94285-kube-api-access-x7lr9" pod "72b4ea44-8863-40e5-baee-a5e1e0c94285" (UID: "72b4ea44-8863-40e5-baee-a5e1e0c94285"). InnerVolumeSpecName "kube-api-access-x7lr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:25.555376 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.554496 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-envoy" pod "72b4ea44-8863-40e5-baee-a5e1e0c94285" (UID: "72b4ea44-8863-40e5-baee-a5e1e0c94285"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:25.562274 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.560998 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-podinfo" pod "72b4ea44-8863-40e5-baee-a5e1e0c94285" (UID: "72b4ea44-8863-40e5-baee-a5e1e0c94285"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:11:25.565604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.565571 5057 generic.go:356] "Generic (PLEG): container finished" podID="72b4ea44-8863-40e5-baee-a5e1e0c94285" containerID="ae3e383566ea7c6451ef12660a6ae7e23e8e30e2ddb69b02bfe85dd52471b02b" exitCode=0 Oct 29 23:11:25.565841 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.565821 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" Oct 29 23:11:25.568163 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.568136 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" event={"ID":"72b4ea44-8863-40e5-baee-a5e1e0c94285","Type":"ContainerDied","Data":"ae3e383566ea7c6451ef12660a6ae7e23e8e30e2ddb69b02bfe85dd52471b02b"} Oct 29 23:11:25.568292 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.568228 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl" event={"ID":"72b4ea44-8863-40e5-baee-a5e1e0c94285","Type":"ContainerDied","Data":"32f50860274d7ba321c8e4074c26d95460a28d82b68d74d9769aa4f96b82542d"} Oct 29 23:11:25.568360 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.568263 5057 scope.go:122] "RemoveContainer" containerID="ae3e383566ea7c6451ef12660a6ae7e23e8e30e2ddb69b02bfe85dd52471b02b" Oct 29 23:11:25.594487 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.594442 5057 scope.go:122] "RemoveContainer" containerID="9f0fd9f5b13f83680835610c4c9f4b45f72d05c529fff22ee16a2fdc83456df4" Oct 29 23:11:25.608084 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.608050 5057 scope.go:122] "RemoveContainer" containerID="ae3e383566ea7c6451ef12660a6ae7e23e8e30e2ddb69b02bfe85dd52471b02b" Oct 29 23:11:25.608569 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:11:25.608533 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae3e383566ea7c6451ef12660a6ae7e23e8e30e2ddb69b02bfe85dd52471b02b\": container with ID starting with ae3e383566ea7c6451ef12660a6ae7e23e8e30e2ddb69b02bfe85dd52471b02b not found: ID does not exist" containerID="ae3e383566ea7c6451ef12660a6ae7e23e8e30e2ddb69b02bfe85dd52471b02b" Oct 29 23:11:25.608694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.608584 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae3e383566ea7c6451ef12660a6ae7e23e8e30e2ddb69b02bfe85dd52471b02b"} err="failed to get container status \"ae3e383566ea7c6451ef12660a6ae7e23e8e30e2ddb69b02bfe85dd52471b02b\": rpc error: code = NotFound desc = could not find container \"ae3e383566ea7c6451ef12660a6ae7e23e8e30e2ddb69b02bfe85dd52471b02b\": container with ID starting with ae3e383566ea7c6451ef12660a6ae7e23e8e30e2ddb69b02bfe85dd52471b02b not found: ID does not exist" Oct 29 23:11:25.608694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.608617 5057 scope.go:122] "RemoveContainer" containerID="9f0fd9f5b13f83680835610c4c9f4b45f72d05c529fff22ee16a2fdc83456df4" Oct 29 23:11:25.608963 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:11:25.608930 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0fd9f5b13f83680835610c4c9f4b45f72d05c529fff22ee16a2fdc83456df4\": container with ID starting with 9f0fd9f5b13f83680835610c4c9f4b45f72d05c529fff22ee16a2fdc83456df4 not found: ID does not exist" containerID="9f0fd9f5b13f83680835610c4c9f4b45f72d05c529fff22ee16a2fdc83456df4" Oct 29 23:11:25.609025 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.608977 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0fd9f5b13f83680835610c4c9f4b45f72d05c529fff22ee16a2fdc83456df4"} err="failed to get container status \"9f0fd9f5b13f83680835610c4c9f4b45f72d05c529fff22ee16a2fdc83456df4\": rpc error: code = NotFound desc = could not find container \"9f0fd9f5b13f83680835610c4c9f4b45f72d05c529fff22ee16a2fdc83456df4\": container with ID starting with 9f0fd9f5b13f83680835610c4c9f4b45f72d05c529fff22ee16a2fdc83456df4 not found: ID does not exist" Oct 29 23:11:25.614334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.614316 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b4ea44-8863-40e5-baee-a5e1e0c94285-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:25.614334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.614335 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:25.614476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.614344 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:25.614476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.614353 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:25.614476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.614365 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:25.614476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.614375 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b4ea44-8863-40e5-baee-a5e1e0c94285-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:25.614476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.614383 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:25.614476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.614391 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7lr9\" (UniqueName: \"kubernetes.io/projected/72b4ea44-8863-40e5-baee-a5e1e0c94285-kube-api-access-x7lr9\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:25.614476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.614400 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b4ea44-8863-40e5-baee-a5e1e0c94285-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:25.637376 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.637349 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl"] Oct 29 23:11:25.643029 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:25.643004 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-85c702-openshift-default-5db5457cb9-8bwbl"] Oct 29 23:11:27.503553 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:27.503456 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b4ea44-8863-40e5-baee-a5e1e0c94285" path="/var/lib/kubelet/pods/72b4ea44-8863-40e5-baee-a5e1e0c94285/volumes" Oct 29 23:11:33.220401 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:33.220366 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql"] Oct 29 23:11:33.220848 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:33.220593 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" podUID="5e20521b-2d52-4c16-bb50-48988a9671bb" containerName="istio-proxy" containerID="cri-o://5b655bc629f6cd2bcadbd224f3e267996aa770b6a8018eecfe5f66fabbabf5b9" gracePeriod=30 Oct 29 23:11:33.936564 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:11:33.936494 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:11:34.023134 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.023095 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh"] Oct 29 23:11:34.023371 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.023324 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh" podUID="59a01498-b3f7-4149-856f-5f3660ec3e94" containerName="payload-processing" containerID="cri-o://244c67f750b18bdf3845030acf2ed932fcef519933e5da0a2a887a2728a583bb" gracePeriod=30 Oct 29 23:11:34.043827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.043489 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd"] Oct 29 23:11:34.044300 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.044247 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd" podUID="fc1022f1-bd79-44de-82e7-58be9da67f5d" containerName="payload-pre-processing" containerID="cri-o://8f64b7092aec3bb693c24d63d60e17e1f9bf101e2a9ae841948a1ef73cbcc6be" gracePeriod=30 Oct 29 23:11:34.615693 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.615174 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd" Oct 29 23:11:34.723171 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.721756 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc1022f1-bd79-44de-82e7-58be9da67f5d-kube-api-access-blgh5\" (UniqueName: \"kubernetes.io/projected/fc1022f1-bd79-44de-82e7-58be9da67f5d-kube-api-access-blgh5\") pod \"fc1022f1-bd79-44de-82e7-58be9da67f5d\" (UID: \"fc1022f1-bd79-44de-82e7-58be9da67f5d\") " Oct 29 23:11:34.723171 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.721925 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1022f1-bd79-44de-82e7-58be9da67f5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1022f1-bd79-44de-82e7-58be9da67f5d-plugins-config-volume\") pod \"fc1022f1-bd79-44de-82e7-58be9da67f5d\" (UID: \"fc1022f1-bd79-44de-82e7-58be9da67f5d\") " Oct 29 23:11:34.724512 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.724167 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1022f1-bd79-44de-82e7-58be9da67f5d-plugins-config-volume" pod "fc1022f1-bd79-44de-82e7-58be9da67f5d" (UID: "fc1022f1-bd79-44de-82e7-58be9da67f5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:34.725411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.724867 5057 generic.go:356] "Generic (PLEG): container finished" podID="fc1022f1-bd79-44de-82e7-58be9da67f5d" containerID="8f64b7092aec3bb693c24d63d60e17e1f9bf101e2a9ae841948a1ef73cbcc6be" exitCode=0 Oct 29 23:11:34.725411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.725059 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd" event={"ID":"fc1022f1-bd79-44de-82e7-58be9da67f5d","Type":"ContainerDied","Data":"8f64b7092aec3bb693c24d63d60e17e1f9bf101e2a9ae841948a1ef73cbcc6be"} Oct 29 23:11:34.725411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.725107 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd" event={"ID":"fc1022f1-bd79-44de-82e7-58be9da67f5d","Type":"ContainerDied","Data":"605ca2d6749994d53a10bf1ce868b0c3dc25240cb565eb0898b6806ebe8b7a8f"} Oct 29 23:11:34.725411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.725138 5057 scope.go:122] "RemoveContainer" containerID="8f64b7092aec3bb693c24d63d60e17e1f9bf101e2a9ae841948a1ef73cbcc6be" Oct 29 23:11:34.726276 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.725870 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd" Oct 29 23:11:34.736695 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.736418 5057 generic.go:356] "Generic (PLEG): container finished" podID="59a01498-b3f7-4149-856f-5f3660ec3e94" containerID="244c67f750b18bdf3845030acf2ed932fcef519933e5da0a2a887a2728a583bb" exitCode=0 Oct 29 23:11:34.736695 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.736642 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh" event={"ID":"59a01498-b3f7-4149-856f-5f3660ec3e94","Type":"ContainerDied","Data":"244c67f750b18bdf3845030acf2ed932fcef519933e5da0a2a887a2728a583bb"} Oct 29 23:11:34.764775 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.764659 5057 scope.go:122] "RemoveContainer" containerID="8f64b7092aec3bb693c24d63d60e17e1f9bf101e2a9ae841948a1ef73cbcc6be" Oct 29 23:11:34.765652 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:11:34.765571 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f64b7092aec3bb693c24d63d60e17e1f9bf101e2a9ae841948a1ef73cbcc6be\": container with ID starting with 8f64b7092aec3bb693c24d63d60e17e1f9bf101e2a9ae841948a1ef73cbcc6be not found: ID does not exist" containerID="8f64b7092aec3bb693c24d63d60e17e1f9bf101e2a9ae841948a1ef73cbcc6be" Oct 29 23:11:34.765652 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.765610 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f64b7092aec3bb693c24d63d60e17e1f9bf101e2a9ae841948a1ef73cbcc6be"} err="failed to get container status \"8f64b7092aec3bb693c24d63d60e17e1f9bf101e2a9ae841948a1ef73cbcc6be\": rpc error: code = NotFound desc = could not find container \"8f64b7092aec3bb693c24d63d60e17e1f9bf101e2a9ae841948a1ef73cbcc6be\": container with ID starting with 8f64b7092aec3bb693c24d63d60e17e1f9bf101e2a9ae841948a1ef73cbcc6be not found: ID does not exist" Oct 29 23:11:34.776474 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.776421 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1022f1-bd79-44de-82e7-58be9da67f5d-kube-api-access-blgh5" pod "fc1022f1-bd79-44de-82e7-58be9da67f5d" (UID: "fc1022f1-bd79-44de-82e7-58be9da67f5d"). InnerVolumeSpecName "kube-api-access-blgh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:34.796586 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.796559 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh" Oct 29 23:11:34.824015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.823981 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59a01498-b3f7-4149-856f-5f3660ec3e94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59a01498-b3f7-4149-856f-5f3660ec3e94-plugins-config-volume\") pod \"59a01498-b3f7-4149-856f-5f3660ec3e94\" (UID: \"59a01498-b3f7-4149-856f-5f3660ec3e94\") " Oct 29 23:11:34.824282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.824230 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a01498-b3f7-4149-856f-5f3660ec3e94-kube-api-access-88hqj\" (UniqueName: \"kubernetes.io/projected/59a01498-b3f7-4149-856f-5f3660ec3e94-kube-api-access-88hqj\") pod \"59a01498-b3f7-4149-856f-5f3660ec3e94\" (UID: \"59a01498-b3f7-4149-856f-5f3660ec3e94\") " Oct 29 23:11:34.824777 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.824745 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a01498-b3f7-4149-856f-5f3660ec3e94-plugins-config-volume" pod "59a01498-b3f7-4149-856f-5f3660ec3e94" (UID: "59a01498-b3f7-4149-856f-5f3660ec3e94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:34.824953 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.824935 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blgh5\" (UniqueName: \"kubernetes.io/projected/fc1022f1-bd79-44de-82e7-58be9da67f5d-kube-api-access-blgh5\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:34.825028 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.824958 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59a01498-b3f7-4149-856f-5f3660ec3e94-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:34.825028 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.824971 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1022f1-bd79-44de-82e7-58be9da67f5d-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:34.826518 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.826492 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a01498-b3f7-4149-856f-5f3660ec3e94-kube-api-access-88hqj" pod "59a01498-b3f7-4149-856f-5f3660ec3e94" (UID: "59a01498-b3f7-4149-856f-5f3660ec3e94"). InnerVolumeSpecName "kube-api-access-88hqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:34.926792 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:34.926756 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88hqj\" (UniqueName: \"kubernetes.io/projected/59a01498-b3f7-4149-856f-5f3660ec3e94-kube-api-access-88hqj\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:35.156407 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:35.156373 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd"] Oct 29 23:11:35.159658 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:35.159633 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-5f97fc8b55-cfbnd"] Oct 29 23:11:35.485598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:35.485565 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc1022f1-bd79-44de-82e7-58be9da67f5d" path="/var/lib/kubelet/pods/fc1022f1-bd79-44de-82e7-58be9da67f5d/volumes" Oct 29 23:11:35.747487 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:35.747391 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh" event={"ID":"59a01498-b3f7-4149-856f-5f3660ec3e94","Type":"ContainerDied","Data":"346a1002dcf275f3b09f491f14900103fbd72c0e85e013b4f9c89893eafe8d3d"} Oct 29 23:11:35.747487 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:35.747458 5057 scope.go:122] "RemoveContainer" containerID="244c67f750b18bdf3845030acf2ed932fcef519933e5da0a2a887a2728a583bb" Oct 29 23:11:35.748028 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:35.747567 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh" Oct 29 23:11:35.807103 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:35.807054 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh"] Oct 29 23:11:35.820488 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:35.820456 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-31d798-fd44db6f6-28vzh"] Oct 29 23:11:37.486035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:37.486002 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59a01498-b3f7-4149-856f-5f3660ec3e94" path="/var/lib/kubelet/pods/59a01498-b3f7-4149-856f-5f3660ec3e94/volumes" Oct 29 23:11:38.016552 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.016514 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:11:38.016709 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.016571 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" podUID="5e20521b-2d52-4c16-bb50-48988a9671bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:11:38.545505 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.545478 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 23:11:38.653040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.652963 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e20521b-2d52-4c16-bb50-48988a9671bb-kube-api-access-66pbp\" (UniqueName: \"kubernetes.io/projected/5e20521b-2d52-4c16-bb50-48988a9671bb-kube-api-access-66pbp\") pod \"5e20521b-2d52-4c16-bb50-48988a9671bb\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " Oct 29 23:11:38.653040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.653012 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-podinfo\") pod \"5e20521b-2d52-4c16-bb50-48988a9671bb\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " Oct 29 23:11:38.653290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.653119 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-data\") pod \"5e20521b-2d52-4c16-bb50-48988a9671bb\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " Oct 29 23:11:38.653290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.653174 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e20521b-2d52-4c16-bb50-48988a9671bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e20521b-2d52-4c16-bb50-48988a9671bb-istiod-ca-cert\") pod \"5e20521b-2d52-4c16-bb50-48988a9671bb\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " Oct 29 23:11:38.653415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.653382 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-envoy\") pod \"5e20521b-2d52-4c16-bb50-48988a9671bb\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " Oct 29 23:11:38.653476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.653437 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-token\" (UniqueName: \"kubernetes.io/projected/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-token\") pod \"5e20521b-2d52-4c16-bb50-48988a9671bb\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " Oct 29 23:11:38.653550 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.653523 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-data" pod "5e20521b-2d52-4c16-bb50-48988a9671bb" (UID: "5e20521b-2d52-4c16-bb50-48988a9671bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:38.653664 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.653638 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e20521b-2d52-4c16-bb50-48988a9671bb-istiod-ca-cert" pod "5e20521b-2d52-4c16-bb50-48988a9671bb" (UID: "5e20521b-2d52-4c16-bb50-48988a9671bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:38.653849 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.653829 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-workload-certs\") pod \"5e20521b-2d52-4c16-bb50-48988a9671bb\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " Oct 29 23:11:38.654000 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.653976 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-credential-socket\") pod \"5e20521b-2d52-4c16-bb50-48988a9671bb\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " Oct 29 23:11:38.654523 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.654065 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-workload-socket\") pod \"5e20521b-2d52-4c16-bb50-48988a9671bb\" (UID: \"5e20521b-2d52-4c16-bb50-48988a9671bb\") " Oct 29 23:11:38.654523 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.654161 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-workload-certs" pod "5e20521b-2d52-4c16-bb50-48988a9671bb" (UID: "5e20521b-2d52-4c16-bb50-48988a9671bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:38.654523 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.654414 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-credential-socket" pod "5e20521b-2d52-4c16-bb50-48988a9671bb" (UID: "5e20521b-2d52-4c16-bb50-48988a9671bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:38.654775 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.654537 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-workload-socket" pod "5e20521b-2d52-4c16-bb50-48988a9671bb" (UID: "5e20521b-2d52-4c16-bb50-48988a9671bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:38.655035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.655012 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:38.655153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.655042 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:38.655153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.655059 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e20521b-2d52-4c16-bb50-48988a9671bb-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:38.655153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.655071 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:38.655153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.655083 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:38.655905 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.655882 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e20521b-2d52-4c16-bb50-48988a9671bb-kube-api-access-66pbp" pod "5e20521b-2d52-4c16-bb50-48988a9671bb" (UID: "5e20521b-2d52-4c16-bb50-48988a9671bb"). InnerVolumeSpecName "kube-api-access-66pbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:38.656001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.655917 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-token" pod "5e20521b-2d52-4c16-bb50-48988a9671bb" (UID: "5e20521b-2d52-4c16-bb50-48988a9671bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:38.656001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.655973 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-envoy" pod "5e20521b-2d52-4c16-bb50-48988a9671bb" (UID: "5e20521b-2d52-4c16-bb50-48988a9671bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:38.656296 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.656270 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-podinfo" pod "5e20521b-2d52-4c16-bb50-48988a9671bb" (UID: "5e20521b-2d52-4c16-bb50-48988a9671bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:11:38.756763 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.756720 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66pbp\" (UniqueName: \"kubernetes.io/projected/5e20521b-2d52-4c16-bb50-48988a9671bb-kube-api-access-66pbp\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:38.756763 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.756753 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:38.756763 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.756765 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:38.756763 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.756774 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e20521b-2d52-4c16-bb50-48988a9671bb-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:38.772781 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.772751 5057 generic.go:356] "Generic (PLEG): container finished" podID="5e20521b-2d52-4c16-bb50-48988a9671bb" containerID="5b655bc629f6cd2bcadbd224f3e267996aa770b6a8018eecfe5f66fabbabf5b9" exitCode=0 Oct 29 23:11:38.772951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.772801 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" Oct 29 23:11:38.772951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.772842 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" event={"ID":"5e20521b-2d52-4c16-bb50-48988a9671bb","Type":"ContainerDied","Data":"5b655bc629f6cd2bcadbd224f3e267996aa770b6a8018eecfe5f66fabbabf5b9"} Oct 29 23:11:38.772951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.772897 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql" event={"ID":"5e20521b-2d52-4c16-bb50-48988a9671bb","Type":"ContainerDied","Data":"5bd9f7f730ce1c9728356405dc48e67559f5de8e54ebc6aa95a9beccb2b95e31"} Oct 29 23:11:38.772951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.772921 5057 scope.go:122] "RemoveContainer" containerID="5b655bc629f6cd2bcadbd224f3e267996aa770b6a8018eecfe5f66fabbabf5b9" Oct 29 23:11:38.787274 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.787251 5057 scope.go:122] "RemoveContainer" containerID="5b655bc629f6cd2bcadbd224f3e267996aa770b6a8018eecfe5f66fabbabf5b9" Oct 29 23:11:38.787861 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:11:38.787831 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b655bc629f6cd2bcadbd224f3e267996aa770b6a8018eecfe5f66fabbabf5b9\": container with ID starting with 5b655bc629f6cd2bcadbd224f3e267996aa770b6a8018eecfe5f66fabbabf5b9 not found: ID does not exist" containerID="5b655bc629f6cd2bcadbd224f3e267996aa770b6a8018eecfe5f66fabbabf5b9" Oct 29 23:11:38.787991 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.787870 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b655bc629f6cd2bcadbd224f3e267996aa770b6a8018eecfe5f66fabbabf5b9"} err="failed to get container status \"5b655bc629f6cd2bcadbd224f3e267996aa770b6a8018eecfe5f66fabbabf5b9\": rpc error: code = NotFound desc = could not find container \"5b655bc629f6cd2bcadbd224f3e267996aa770b6a8018eecfe5f66fabbabf5b9\": container with ID starting with 5b655bc629f6cd2bcadbd224f3e267996aa770b6a8018eecfe5f66fabbabf5b9 not found: ID does not exist" Oct 29 23:11:38.831606 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.831566 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql"] Oct 29 23:11:38.838698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:38.838666 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-sl6ql"] Oct 29 23:11:39.484324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:39.484292 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e20521b-2d52-4c16-bb50-48988a9671bb" path="/var/lib/kubelet/pods/5e20521b-2d52-4c16-bb50-48988a9671bb/volumes" Oct 29 23:11:42.516902 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:42.516871 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh"] Oct 29 23:11:42.517438 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:42.517080 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" podUID="d35fd058-ef20-4d71-bd81-0ecf36bef728" containerName="istio-proxy" containerID="cri-o://8effa6f953946c40884fd16bfff84c42e653ec6a0e53801f1afd1fc7e0355e9c" gracePeriod=30 Oct 29 23:11:43.276287 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:43.276249 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:11:43.276467 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:43.276347 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" podUID="d35fd058-ef20-4d71-bd81-0ecf36bef728" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:11:44.542921 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.542880 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65"] Oct 29 23:11:44.544356 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544328 5057 cpu_manager.go:414] "containerMap: removing container" podUID="59a01498-b3f7-4149-856f-5f3660ec3e94" containerName="payload-processing" Oct 29 23:11:44.544356 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544354 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a01498-b3f7-4149-856f-5f3660ec3e94" containerName="payload-processing" Oct 29 23:11:44.544546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544385 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" containerName="istio-proxy" Oct 29 23:11:44.544546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544395 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" containerName="istio-proxy" Oct 29 23:11:44.544546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544412 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5e20521b-2d52-4c16-bb50-48988a9671bb" containerName="istio-proxy" Oct 29 23:11:44.544546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544422 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e20521b-2d52-4c16-bb50-48988a9671bb" containerName="istio-proxy" Oct 29 23:11:44.544546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544439 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" containerName="istio-proxy" Oct 29 23:11:44.544546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544448 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" containerName="istio-proxy" Oct 29 23:11:44.544546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544461 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a6e31a55-feaa-4b94-bb1f-5f7758706537" containerName="payload-pre-processing" Oct 29 23:11:44.544546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544470 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e31a55-feaa-4b94-bb1f-5f7758706537" containerName="payload-pre-processing" Oct 29 23:11:44.544546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544488 5057 cpu_manager.go:414] "containerMap: removing container" podUID="72b4ea44-8863-40e5-baee-a5e1e0c94285" containerName="istio-proxy" Oct 29 23:11:44.544546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544497 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b4ea44-8863-40e5-baee-a5e1e0c94285" containerName="istio-proxy" Oct 29 23:11:44.544546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544519 5057 cpu_manager.go:414] "containerMap: removing container" podUID="884eed63-5693-41b6-81c6-0d4cd1abdc4c" containerName="payload-processing" Oct 29 23:11:44.544546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544528 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884eed63-5693-41b6-81c6-0d4cd1abdc4c" containerName="payload-processing" Oct 29 23:11:44.544546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544548 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e7897e3b-8c0e-45be-a3b1-ad5ccf299283" containerName="payload-pre-processing" Oct 29 23:11:44.546549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544557 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7897e3b-8c0e-45be-a3b1-ad5ccf299283" containerName="payload-pre-processing" Oct 29 23:11:44.546549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544575 5057 cpu_manager.go:414] "containerMap: removing container" podUID="cdf4f520-c1fb-4178-9685-2ddf12262613" containerName="payload-processing" Oct 29 23:11:44.546549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544583 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdf4f520-c1fb-4178-9685-2ddf12262613" containerName="payload-processing" Oct 29 23:11:44.546549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544599 5057 cpu_manager.go:414] "containerMap: removing container" podUID="72b4ea44-8863-40e5-baee-a5e1e0c94285" containerName="istio-proxy" Oct 29 23:11:44.546549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544607 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b4ea44-8863-40e5-baee-a5e1e0c94285" containerName="istio-proxy" Oct 29 23:11:44.546549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544624 5057 cpu_manager.go:414] "containerMap: removing container" podUID="fc1022f1-bd79-44de-82e7-58be9da67f5d" containerName="payload-pre-processing" Oct 29 23:11:44.546549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544633 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc1022f1-bd79-44de-82e7-58be9da67f5d" containerName="payload-pre-processing" Oct 29 23:11:44.546549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544661 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5409eeb6-ca88-4c97-9ee9-16449a4f9dd6" containerName="maas-api" Oct 29 23:11:44.546549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544669 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5409eeb6-ca88-4c97-9ee9-16449a4f9dd6" containerName="maas-api" Oct 29 23:11:44.546549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544849 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e20521b-2d52-4c16-bb50-48988a9671bb" containerName="istio-proxy" Oct 29 23:11:44.546549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544867 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" containerName="istio-proxy" Oct 29 23:11:44.546549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544877 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e31a55-feaa-4b94-bb1f-5f7758706537" containerName="payload-pre-processing" Oct 29 23:11:44.546549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544888 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b4ea44-8863-40e5-baee-a5e1e0c94285" containerName="istio-proxy" Oct 29 23:11:44.546549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544898 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="884eed63-5693-41b6-81c6-0d4cd1abdc4c" containerName="payload-processing" Oct 29 23:11:44.546549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544910 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7897e3b-8c0e-45be-a3b1-ad5ccf299283" containerName="payload-pre-processing" Oct 29 23:11:44.546549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544921 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdf4f520-c1fb-4178-9685-2ddf12262613" containerName="payload-processing" Oct 29 23:11:44.546549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544931 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc1022f1-bd79-44de-82e7-58be9da67f5d" containerName="payload-pre-processing" Oct 29 23:11:44.546549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544944 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5409eeb6-ca88-4c97-9ee9-16449a4f9dd6" containerName="maas-api" Oct 29 23:11:44.546549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.544958 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a01498-b3f7-4149-856f-5f3660ec3e94" containerName="payload-processing" Oct 29 23:11:44.551117 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.551085 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.570224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.570167 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-96cd49-openshift-default-dockercfg-959wn\"" Oct 29 23:11:44.589998 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.589959 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65"] Oct 29 23:11:44.686963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.686926 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-workload-certs\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.687147 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.686976 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-podinfo\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.687147 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.687041 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istiod-ca-cert\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.687147 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.687112 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-credential-socket\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.687380 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.687240 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-workload-socket\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.687380 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.687324 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-envoy\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.687491 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.687407 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-token\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.687491 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.687459 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6zdq\" (UniqueName: \"kubernetes.io/projected/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-kube-api-access-j6zdq\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.687604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.687505 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-data\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.811258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.801032 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-workload-socket\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.811258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.801154 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-envoy\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.811258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.801262 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-token\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.811258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.801318 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6zdq\" (UniqueName: \"kubernetes.io/projected/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-kube-api-access-j6zdq\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.811258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.801379 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-data\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.811258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.801429 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-workload-certs\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.811258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.801484 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-podinfo\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.811258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.801536 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istiod-ca-cert\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.811258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.801576 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-credential-socket\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.811258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.802065 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-credential-socket\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.811258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.802360 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-workload-socket\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.811258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.809533 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-data\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.817210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.815015 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istiod-ca-cert\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.817210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.815317 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-workload-certs\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.817723 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.817700 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-envoy\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.843837 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.843790 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-podinfo\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.860205 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.857968 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6zdq\" (UniqueName: \"kubernetes.io/projected/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-kube-api-access-j6zdq\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:44.916213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:44.900954 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-token\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:45.170554 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:45.170470 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:45.496282 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:11:45.496247 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ecdcde5_2bbb_42cc_b89e_2ff1e9da5276.slice/crio-5ad2a0fa4a5e8d8cc4ee405ccf112d3fe2327e5f2d02cea4c42919d78de0347d WatchSource:0}: Error finding container 5ad2a0fa4a5e8d8cc4ee405ccf112d3fe2327e5f2d02cea4c42919d78de0347d: Status 404 returned error can't find the container with id 5ad2a0fa4a5e8d8cc4ee405ccf112d3fe2327e5f2d02cea4c42919d78de0347d Oct 29 23:11:45.497755 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:45.497729 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65"] Oct 29 23:11:45.498910 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:45.498880 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:11:45.498979 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:45.498946 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:11:45.499016 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:45.498977 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:11:45.875952 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:45.875852 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" event={"ID":"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276","Type":"ContainerStarted","Data":"1c2aa93d278358cb53967ac1f42e3068758e189c36e5c58fb78d1c8d8c5a6006"} Oct 29 23:11:45.876472 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:45.875958 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" event={"ID":"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276","Type":"ContainerStarted","Data":"5ad2a0fa4a5e8d8cc4ee405ccf112d3fe2327e5f2d02cea4c42919d78de0347d"} Oct 29 23:11:45.929518 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:45.929453 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" podStartSLOduration=1.9294327180000002 podStartE2EDuration="1.929432718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:45.924657837 +0000 UTC m=+1752.576156370" watchObservedRunningTime="2026-10-29 23:11:45.929432718 +0000 UTC m=+1752.580931249" Oct 29 23:11:46.171630 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:46.171522 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:46.177197 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:46.177160 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:46.884049 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:46.884006 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:46.885071 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:46.885048 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:11:47.834778 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.834442 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:11:47.862591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.862551 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-token\" (UniqueName: \"kubernetes.io/projected/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-token\") pod \"d35fd058-ef20-4d71-bd81-0ecf36bef728\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " Oct 29 23:11:47.862795 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.862675 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-credential-socket\") pod \"d35fd058-ef20-4d71-bd81-0ecf36bef728\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " Oct 29 23:11:47.862865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.862790 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35fd058-ef20-4d71-bd81-0ecf36bef728-kube-api-access-btpd7\" (UniqueName: \"kubernetes.io/projected/d35fd058-ef20-4d71-bd81-0ecf36bef728-kube-api-access-btpd7\") pod \"d35fd058-ef20-4d71-bd81-0ecf36bef728\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " Oct 29 23:11:47.862865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.862834 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-envoy\") pod \"d35fd058-ef20-4d71-bd81-0ecf36bef728\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " Oct 29 23:11:47.862977 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.862864 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-workload-socket\") pod \"d35fd058-ef20-4d71-bd81-0ecf36bef728\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " Oct 29 23:11:47.862977 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.862895 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-workload-certs\") pod \"d35fd058-ef20-4d71-bd81-0ecf36bef728\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " Oct 29 23:11:47.863090 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.862977 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d35fd058-ef20-4d71-bd81-0ecf36bef728-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d35fd058-ef20-4d71-bd81-0ecf36bef728-istiod-ca-cert\") pod \"d35fd058-ef20-4d71-bd81-0ecf36bef728\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " Oct 29 23:11:47.863090 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.863032 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-podinfo\") pod \"d35fd058-ef20-4d71-bd81-0ecf36bef728\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " Oct 29 23:11:47.863090 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.863063 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-data\") pod \"d35fd058-ef20-4d71-bd81-0ecf36bef728\" (UID: \"d35fd058-ef20-4d71-bd81-0ecf36bef728\") " Oct 29 23:11:47.863277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.863118 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-workload-socket" pod "d35fd058-ef20-4d71-bd81-0ecf36bef728" (UID: "d35fd058-ef20-4d71-bd81-0ecf36bef728"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:47.863277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.863205 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-workload-certs" pod "d35fd058-ef20-4d71-bd81-0ecf36bef728" (UID: "d35fd058-ef20-4d71-bd81-0ecf36bef728"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:47.863277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.862991 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-credential-socket" pod "d35fd058-ef20-4d71-bd81-0ecf36bef728" (UID: "d35fd058-ef20-4d71-bd81-0ecf36bef728"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:47.863459 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.863434 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35fd058-ef20-4d71-bd81-0ecf36bef728-istiod-ca-cert" pod "d35fd058-ef20-4d71-bd81-0ecf36bef728" (UID: "d35fd058-ef20-4d71-bd81-0ecf36bef728"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:47.863961 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.863927 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-data" pod "d35fd058-ef20-4d71-bd81-0ecf36bef728" (UID: "d35fd058-ef20-4d71-bd81-0ecf36bef728"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:47.864252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.864181 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:47.864252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.864244 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:47.864431 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.864261 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:47.864431 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.864275 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d35fd058-ef20-4d71-bd81-0ecf36bef728-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:47.865258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.865233 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-token" pod "d35fd058-ef20-4d71-bd81-0ecf36bef728" (UID: "d35fd058-ef20-4d71-bd81-0ecf36bef728"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:47.867231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.865679 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-envoy" pod "d35fd058-ef20-4d71-bd81-0ecf36bef728" (UID: "d35fd058-ef20-4d71-bd81-0ecf36bef728"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:47.867231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.866358 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35fd058-ef20-4d71-bd81-0ecf36bef728-kube-api-access-btpd7" pod "d35fd058-ef20-4d71-bd81-0ecf36bef728" (UID: "d35fd058-ef20-4d71-bd81-0ecf36bef728"). InnerVolumeSpecName "kube-api-access-btpd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:47.867231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.866825 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-podinfo" pod "d35fd058-ef20-4d71-bd81-0ecf36bef728" (UID: "d35fd058-ef20-4d71-bd81-0ecf36bef728"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:11:47.872572 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.872550 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf"] Oct 29 23:11:47.893040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.893014 5057 generic.go:356] "Generic (PLEG): container finished" podID="d35fd058-ef20-4d71-bd81-0ecf36bef728" containerID="8effa6f953946c40884fd16bfff84c42e653ec6a0e53801f1afd1fc7e0355e9c" exitCode=0 Oct 29 23:11:47.893423 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.893064 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" Oct 29 23:11:47.893423 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.893099 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" event={"ID":"d35fd058-ef20-4d71-bd81-0ecf36bef728","Type":"ContainerDied","Data":"8effa6f953946c40884fd16bfff84c42e653ec6a0e53801f1afd1fc7e0355e9c"} Oct 29 23:11:47.893423 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.893150 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh" event={"ID":"d35fd058-ef20-4d71-bd81-0ecf36bef728","Type":"ContainerDied","Data":"fe9c171cafa6ce4449505b58a0962769365a955a9071b4aacbc6057f5d83c22a"} Oct 29 23:11:47.893423 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.893172 5057 scope.go:122] "RemoveContainer" containerID="8effa6f953946c40884fd16bfff84c42e653ec6a0e53801f1afd1fc7e0355e9c" Oct 29 23:11:47.907559 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.907248 5057 scope.go:122] "RemoveContainer" containerID="9b043a9d42fbfc35ec3d5d792e10dc334e5bd73f1070aa14c95ff1e197eacbaf" Oct 29 23:11:47.918094 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.918073 5057 scope.go:122] "RemoveContainer" containerID="8effa6f953946c40884fd16bfff84c42e653ec6a0e53801f1afd1fc7e0355e9c" Oct 29 23:11:47.918396 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:11:47.918370 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8effa6f953946c40884fd16bfff84c42e653ec6a0e53801f1afd1fc7e0355e9c\": container with ID starting with 8effa6f953946c40884fd16bfff84c42e653ec6a0e53801f1afd1fc7e0355e9c not found: ID does not exist" containerID="8effa6f953946c40884fd16bfff84c42e653ec6a0e53801f1afd1fc7e0355e9c" Oct 29 23:11:47.918492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.918404 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8effa6f953946c40884fd16bfff84c42e653ec6a0e53801f1afd1fc7e0355e9c"} err="failed to get container status \"8effa6f953946c40884fd16bfff84c42e653ec6a0e53801f1afd1fc7e0355e9c\": rpc error: code = NotFound desc = could not find container \"8effa6f953946c40884fd16bfff84c42e653ec6a0e53801f1afd1fc7e0355e9c\": container with ID starting with 8effa6f953946c40884fd16bfff84c42e653ec6a0e53801f1afd1fc7e0355e9c not found: ID does not exist" Oct 29 23:11:47.918492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.918421 5057 scope.go:122] "RemoveContainer" containerID="9b043a9d42fbfc35ec3d5d792e10dc334e5bd73f1070aa14c95ff1e197eacbaf" Oct 29 23:11:47.918680 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:11:47.918660 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b043a9d42fbfc35ec3d5d792e10dc334e5bd73f1070aa14c95ff1e197eacbaf\": container with ID starting with 9b043a9d42fbfc35ec3d5d792e10dc334e5bd73f1070aa14c95ff1e197eacbaf not found: ID does not exist" containerID="9b043a9d42fbfc35ec3d5d792e10dc334e5bd73f1070aa14c95ff1e197eacbaf" Oct 29 23:11:47.918745 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.918685 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b043a9d42fbfc35ec3d5d792e10dc334e5bd73f1070aa14c95ff1e197eacbaf"} err="failed to get container status \"9b043a9d42fbfc35ec3d5d792e10dc334e5bd73f1070aa14c95ff1e197eacbaf\": rpc error: code = NotFound desc = could not find container \"9b043a9d42fbfc35ec3d5d792e10dc334e5bd73f1070aa14c95ff1e197eacbaf\": container with ID starting with 9b043a9d42fbfc35ec3d5d792e10dc334e5bd73f1070aa14c95ff1e197eacbaf not found: ID does not exist" Oct 29 23:11:47.971939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.971850 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:47.973007 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.972978 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:47.973241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.973227 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:47.973362 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.973350 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btpd7\" (UniqueName: \"kubernetes.io/projected/d35fd058-ef20-4d71-bd81-0ecf36bef728-kube-api-access-btpd7\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:47.973448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:47.973437 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d35fd058-ef20-4d71-bd81-0ecf36bef728-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:48.019128 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:48.019096 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh"] Oct 29 23:11:48.026003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:48.025966 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-31d798-openshift-default-68bf54b8b9-hc2vh"] Oct 29 23:11:49.482891 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:49.482859 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d35fd058-ef20-4d71-bd81-0ecf36bef728" path="/var/lib/kubelet/pods/d35fd058-ef20-4d71-bd81-0ecf36bef728/volumes" Oct 29 23:11:50.165770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.165738 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k"] Oct 29 23:11:50.166543 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.166522 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d35fd058-ef20-4d71-bd81-0ecf36bef728" containerName="istio-proxy" Oct 29 23:11:50.166543 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.166542 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35fd058-ef20-4d71-bd81-0ecf36bef728" containerName="istio-proxy" Oct 29 23:11:50.166761 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.166596 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d35fd058-ef20-4d71-bd81-0ecf36bef728" containerName="istio-proxy" Oct 29 23:11:50.166761 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.166602 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35fd058-ef20-4d71-bd81-0ecf36bef728" containerName="istio-proxy" Oct 29 23:11:50.166761 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.166738 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b4ea44-8863-40e5-baee-a5e1e0c94285" containerName="istio-proxy" Oct 29 23:11:50.166929 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.166763 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35fd058-ef20-4d71-bd81-0ecf36bef728" containerName="istio-proxy" Oct 29 23:11:50.166929 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.166778 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35fd058-ef20-4d71-bd81-0ecf36bef728" containerName="istio-proxy" Oct 29 23:11:50.166929 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.166785 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f96e8f2-d75a-4a6d-a3a8-08f7f0007a33" containerName="istio-proxy" Oct 29 23:11:50.172397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.172376 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k" Oct 29 23:11:50.193122 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.193096 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k"] Oct 29 23:11:50.202853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.202817 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-96cd49\"" Oct 29 23:11:50.202853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.202834 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-96cd49-dockercfg-jvn7p\"" Oct 29 23:11:50.426130 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.410920 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k\" (UID: \"3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k" Oct 29 23:11:50.426130 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.411133 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxkc8\" (UniqueName: \"kubernetes.io/projected/3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6-kube-api-access-zxkc8\") pod \"payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k\" (UID: \"3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k" Oct 29 23:11:50.532207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.525584 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k\" (UID: \"3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k" Oct 29 23:11:50.532207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.525812 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxkc8\" (UniqueName: \"kubernetes.io/projected/3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6-kube-api-access-zxkc8\") pod \"payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k\" (UID: \"3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k" Oct 29 23:11:50.540309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.532906 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k\" (UID: \"3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k" Oct 29 23:11:50.541957 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.541925 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm"] Oct 29 23:11:50.554205 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.550985 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm"] Oct 29 23:11:50.554205 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.551103 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm" Oct 29 23:11:50.565800 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.565767 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxkc8\" (UniqueName: \"kubernetes.io/projected/3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6-kube-api-access-zxkc8\") pod \"payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k\" (UID: \"3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k" Oct 29 23:11:50.627171 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.627139 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bgjb\" (UniqueName: \"kubernetes.io/projected/4528935e-c7f6-4c37-a477-2e006cd53fb3-kube-api-access-8bgjb\") pod \"payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm\" (UID: \"4528935e-c7f6-4c37-a477-2e006cd53fb3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm" Oct 29 23:11:50.627369 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.627351 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4528935e-c7f6-4c37-a477-2e006cd53fb3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm\" (UID: \"4528935e-c7f6-4c37-a477-2e006cd53fb3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm" Oct 29 23:11:50.729261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.729145 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bgjb\" (UniqueName: \"kubernetes.io/projected/4528935e-c7f6-4c37-a477-2e006cd53fb3-kube-api-access-8bgjb\") pod \"payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm\" (UID: \"4528935e-c7f6-4c37-a477-2e006cd53fb3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm" Oct 29 23:11:50.729435 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.729385 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4528935e-c7f6-4c37-a477-2e006cd53fb3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm\" (UID: \"4528935e-c7f6-4c37-a477-2e006cd53fb3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm" Oct 29 23:11:50.730074 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.730048 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4528935e-c7f6-4c37-a477-2e006cd53fb3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm\" (UID: \"4528935e-c7f6-4c37-a477-2e006cd53fb3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm" Oct 29 23:11:50.754519 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.754479 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bgjb\" (UniqueName: \"kubernetes.io/projected/4528935e-c7f6-4c37-a477-2e006cd53fb3-kube-api-access-8bgjb\") pod \"payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm\" (UID: \"4528935e-c7f6-4c37-a477-2e006cd53fb3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm" Oct 29 23:11:50.787304 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.787271 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k" Oct 29 23:11:50.874466 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:50.871428 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm" Oct 29 23:11:51.098353 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:11:51.098321 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae9bc9b_b7ad_4b9b_9d9f_d2edae6d09c6.slice/crio-274d2536e3b43ea2d4aa4d591c9199db557274e4a3dc84c516fd4bf22979d155 WatchSource:0}: Error finding container 274d2536e3b43ea2d4aa4d591c9199db557274e4a3dc84c516fd4bf22979d155: Status 404 returned error can't find the container with id 274d2536e3b43ea2d4aa4d591c9199db557274e4a3dc84c516fd4bf22979d155 Oct 29 23:11:51.099001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:51.098972 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k"] Oct 29 23:11:51.218646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:51.218608 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm"] Oct 29 23:11:51.219049 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:11:51.219016 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4528935e_c7f6_4c37_a477_2e006cd53fb3.slice/crio-14a57b58f09af3f0cc8798c96f8bc65d1b4795744c867b88815fe999ed8c937e WatchSource:0}: Error finding container 14a57b58f09af3f0cc8798c96f8bc65d1b4795744c867b88815fe999ed8c937e: Status 404 returned error can't find the container with id 14a57b58f09af3f0cc8798c96f8bc65d1b4795744c867b88815fe999ed8c937e Oct 29 23:11:51.931881 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:51.931840 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm" event={"ID":"4528935e-c7f6-4c37-a477-2e006cd53fb3","Type":"ContainerStarted","Data":"96a680cf042ae9437491fe29d3946401d6d47304da7e9546ce366833d20b08e9"} Oct 29 23:11:51.932412 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:51.931908 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm" Oct 29 23:11:51.932412 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:51.931932 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm" event={"ID":"4528935e-c7f6-4c37-a477-2e006cd53fb3","Type":"ContainerStarted","Data":"14a57b58f09af3f0cc8798c96f8bc65d1b4795744c867b88815fe999ed8c937e"} Oct 29 23:11:51.933751 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:51.933718 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k" event={"ID":"3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6","Type":"ContainerStarted","Data":"3dbb0b88993c1220c4f7eb32a33308c1636dd25a330ba94cb5efaafc1e239c67"} Oct 29 23:11:51.933882 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:51.933776 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k" event={"ID":"3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6","Type":"ContainerStarted","Data":"274d2536e3b43ea2d4aa4d591c9199db557274e4a3dc84c516fd4bf22979d155"} Oct 29 23:11:51.933882 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:51.933803 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k" Oct 29 23:11:51.979533 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:51.979480 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm" podStartSLOduration=1.979466972 podStartE2EDuration="1.979466972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:51.97607536 +0000 UTC m=+1758.627573890" watchObservedRunningTime="2026-10-29 23:11:51.979466972 +0000 UTC m=+1758.630965559" Oct 29 23:11:52.022837 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:52.022774 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k" podStartSLOduration=2.0227545 podStartE2EDuration="2.0227545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:52.020267144 +0000 UTC m=+1758.671765675" watchObservedRunningTime="2026-10-29 23:11:52.0227545 +0000 UTC m=+1758.674253030" Oct 29 23:11:52.873309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:52.873233 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf" podUID="f42728ab-5de3-4e1d-b388-6e5aa5e21206" containerName="payload-processing" containerID="cri-o://b6a8f956a2a47164b9184b54740ee52ffa105983400afeec96ba78e9369a197c" gracePeriod=25 Oct 29 23:11:53.320082 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:53.320053 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf" Oct 29 23:11:53.404167 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:53.404137 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f42728ab-5de3-4e1d-b388-6e5aa5e21206-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42728ab-5de3-4e1d-b388-6e5aa5e21206-plugins-config-volume\") pod \"f42728ab-5de3-4e1d-b388-6e5aa5e21206\" (UID: \"f42728ab-5de3-4e1d-b388-6e5aa5e21206\") " Oct 29 23:11:53.404371 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:53.404342 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42728ab-5de3-4e1d-b388-6e5aa5e21206-kube-api-access-lg5ht\" (UniqueName: \"kubernetes.io/projected/f42728ab-5de3-4e1d-b388-6e5aa5e21206-kube-api-access-lg5ht\") pod \"f42728ab-5de3-4e1d-b388-6e5aa5e21206\" (UID: \"f42728ab-5de3-4e1d-b388-6e5aa5e21206\") " Oct 29 23:11:53.404728 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:53.404701 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42728ab-5de3-4e1d-b388-6e5aa5e21206-plugins-config-volume" pod "f42728ab-5de3-4e1d-b388-6e5aa5e21206" (UID: "f42728ab-5de3-4e1d-b388-6e5aa5e21206"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:53.405116 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:53.405092 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42728ab-5de3-4e1d-b388-6e5aa5e21206-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:53.406680 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:53.406614 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42728ab-5de3-4e1d-b388-6e5aa5e21206-kube-api-access-lg5ht" pod "f42728ab-5de3-4e1d-b388-6e5aa5e21206" (UID: "f42728ab-5de3-4e1d-b388-6e5aa5e21206"). InnerVolumeSpecName "kube-api-access-lg5ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:53.506425 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:53.506387 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg5ht\" (UniqueName: \"kubernetes.io/projected/f42728ab-5de3-4e1d-b388-6e5aa5e21206-kube-api-access-lg5ht\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:11:53.953772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:53.953739 5057 generic.go:356] "Generic (PLEG): container finished" podID="f42728ab-5de3-4e1d-b388-6e5aa5e21206" containerID="b6a8f956a2a47164b9184b54740ee52ffa105983400afeec96ba78e9369a197c" exitCode=0 Oct 29 23:11:53.953973 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:53.953806 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf" Oct 29 23:11:53.953973 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:53.953822 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf" event={"ID":"f42728ab-5de3-4e1d-b388-6e5aa5e21206","Type":"ContainerDied","Data":"b6a8f956a2a47164b9184b54740ee52ffa105983400afeec96ba78e9369a197c"} Oct 29 23:11:53.953973 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:53.953891 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf" event={"ID":"f42728ab-5de3-4e1d-b388-6e5aa5e21206","Type":"ContainerDied","Data":"da019f4da2681f45347f344218aa75732af728b2a34dc4575c05048dd3159bae"} Oct 29 23:11:53.953973 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:53.953921 5057 scope.go:122] "RemoveContainer" containerID="b6a8f956a2a47164b9184b54740ee52ffa105983400afeec96ba78e9369a197c" Oct 29 23:11:53.965838 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:53.965814 5057 scope.go:122] "RemoveContainer" containerID="b6a8f956a2a47164b9184b54740ee52ffa105983400afeec96ba78e9369a197c" Oct 29 23:11:53.966177 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:11:53.966157 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a8f956a2a47164b9184b54740ee52ffa105983400afeec96ba78e9369a197c\": container with ID starting with b6a8f956a2a47164b9184b54740ee52ffa105983400afeec96ba78e9369a197c not found: ID does not exist" containerID="b6a8f956a2a47164b9184b54740ee52ffa105983400afeec96ba78e9369a197c" Oct 29 23:11:53.966263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:53.966198 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a8f956a2a47164b9184b54740ee52ffa105983400afeec96ba78e9369a197c"} err="failed to get container status \"b6a8f956a2a47164b9184b54740ee52ffa105983400afeec96ba78e9369a197c\": rpc error: code = NotFound desc = could not find container \"b6a8f956a2a47164b9184b54740ee52ffa105983400afeec96ba78e9369a197c\": container with ID starting with b6a8f956a2a47164b9184b54740ee52ffa105983400afeec96ba78e9369a197c not found: ID does not exist" Oct 29 23:11:54.013918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:54.013882 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf"] Oct 29 23:11:54.027505 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:54.027475 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f2c8d3-86c9cd4b9b-mrnvf"] Oct 29 23:11:55.342250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.338280 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9"] Oct 29 23:11:55.342250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.339426 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f42728ab-5de3-4e1d-b388-6e5aa5e21206" containerName="payload-processing" Oct 29 23:11:55.342250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.339446 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42728ab-5de3-4e1d-b388-6e5aa5e21206" containerName="payload-processing" Oct 29 23:11:55.342250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.339608 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42728ab-5de3-4e1d-b388-6e5aa5e21206" containerName="payload-processing" Oct 29 23:11:55.345216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.345180 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.363459 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.363414 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-26a2a4-openshift-default-dockercfg-c8jxz\"" Oct 29 23:11:55.384120 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.384087 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9"] Oct 29 23:11:55.443345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.443303 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istiod-ca-cert\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.443692 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.443668 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-workload-socket\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.443828 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.443813 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-credential-socket\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.443956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.443943 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-workload-certs\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.444064 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.444049 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-token\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.444380 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.444360 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-podinfo\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.444544 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.444528 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6mvx\" (UniqueName: \"kubernetes.io/projected/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-kube-api-access-s6mvx\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.444772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.444753 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-data\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.444933 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.444916 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-envoy\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.549223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.549161 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-data\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.549521 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.549497 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-envoy\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.549685 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.549668 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istiod-ca-cert\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.549838 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.549824 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-workload-socket\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.549964 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.549949 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-credential-socket\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.550107 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.550093 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-workload-certs\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.550266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.550250 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-token\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.563211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.550544 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-podinfo\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.565132 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.565102 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6mvx\" (UniqueName: \"kubernetes.io/projected/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-kube-api-access-s6mvx\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.565584 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.564745 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-data\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.565758 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.564533 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-workload-socket\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.566451 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.566427 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-workload-certs\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.566834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.566814 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-credential-socket\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.567758 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.567736 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istiod-ca-cert\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.586765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.586729 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f42728ab-5de3-4e1d-b388-6e5aa5e21206" path="/var/lib/kubelet/pods/f42728ab-5de3-4e1d-b388-6e5aa5e21206/volumes" Oct 29 23:11:55.632652 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.621170 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-envoy\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.676155 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.674923 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-token\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.676155 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.675702 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6mvx\" (UniqueName: \"kubernetes.io/projected/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-kube-api-access-s6mvx\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.683000 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.682944 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-podinfo\") pod \"e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:55.965008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:55.963881 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:56.427791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:56.427758 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9"] Oct 29 23:11:56.428322 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:11:56.428290 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0197f2_eb04_41dc_8a00_a3f97ee557d3.slice/crio-e22a909c4a871dee72d637f3020ce7c597313743d178001cb0e178b2603db3d5 WatchSource:0}: Error finding container e22a909c4a871dee72d637f3020ce7c597313743d178001cb0e178b2603db3d5: Status 404 returned error can't find the container with id e22a909c4a871dee72d637f3020ce7c597313743d178001cb0e178b2603db3d5 Oct 29 23:11:56.431242 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:56.431202 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:11:56.431366 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:56.431272 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:11:56.431366 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:56.431301 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:11:57.093216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:57.087826 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" event={"ID":"9c0197f2-eb04-41dc-8a00-a3f97ee557d3","Type":"ContainerStarted","Data":"01e4a75d65e031938da18a02fab0cc284eea8825272860d642633b41a7516a67"} Oct 29 23:11:57.093216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:57.087897 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" event={"ID":"9c0197f2-eb04-41dc-8a00-a3f97ee557d3","Type":"ContainerStarted","Data":"e22a909c4a871dee72d637f3020ce7c597313743d178001cb0e178b2603db3d5"} Oct 29 23:11:57.170790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:57.165883 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" podStartSLOduration=2.165863888 podStartE2EDuration="2.165863888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:57.164375754 +0000 UTC m=+1763.815874289" watchObservedRunningTime="2026-10-29 23:11:57.165863888 +0000 UTC m=+1763.817362418" Oct 29 23:11:57.986210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:57.985405 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:58.060815 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:58.060773 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:11:58.241261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:58.240502 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:59.159206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:59.147912 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:11:59.168211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:11:59.163584 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:12:01.138356 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:01.138294 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:01.912347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:01.912290 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:02.035237 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:02.034376 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:02.144436 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:02.144386 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 23:12:02.166656 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:02.166543 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:12:02.698211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:02.694265 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:02.960363 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:02.925640 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:02.960363 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:02.927431 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:12:02.995770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:02.995736 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm" Oct 29 23:12:02.997180 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:02.997157 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k" Oct 29 23:12:03.046406 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:03.046353 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:03.062221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:03.054464 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:12:03.188387 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:03.161634 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:03.735212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:03.706234 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 23:12:04.064291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:04.044052 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:12:04.064291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:04.048342 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:04.213280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:04.186943 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:12:04.217040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:04.215076 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 23:12:04.718311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:04.718264 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:04.730354 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:04.719433 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:05.012525 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:12:05.012419 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:12:05.113085 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:05.113038 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:05.150334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:05.150295 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:12:05.252597 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:05.252550 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:05.286298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:05.256056 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx"] Oct 29 23:12:05.286298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:05.279467 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 23:12:05.287428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:05.286764 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" podUID="f525a668-448c-44ed-8fac-21db37da4c29" containerName="istio-proxy" containerID="cri-o://d9cb9b5f8f27dafff60b70ac0c64a0f3ff6446c7ba21c566e0ffe1d178fa4d18" gracePeriod=30 Oct 29 23:12:05.764280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:05.743613 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:05.989392 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:05.989333 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded" Oct 29 23:12:06.126261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:06.119318 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 23:12:06.173400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:06.173344 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:12:06.316696 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:06.316644 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:06.746008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:06.745955 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:06.814343 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:12:06.811942 5057 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.121s" Oct 29 23:12:07.128268 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:07.119469 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 23:12:07.192213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:07.179646 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:07.326057 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:07.326016 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:07.688824 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:07.687848 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:12:07.688824 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:07.687907 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:12:07.692338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:07.691621 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:12:07.728229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:07.727071 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:12:10.087211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.085248 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw"] Oct 29 23:12:10.198215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.182279 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw"] Oct 29 23:12:10.198215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.182433 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.202208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.200525 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-b73a75-openshift-default-dockercfg-vb8t8\"" Oct 29 23:12:10.317222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.314378 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-envoy\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.317222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.314434 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-podinfo\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.317222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.314619 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-token\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.317222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.314654 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-workload-socket\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.317222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.314712 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-workload-certs\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.317222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.314759 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7tjn\" (UniqueName: \"kubernetes.io/projected/b783e0d4-8272-4c3e-b046-0d6a70160fea-kube-api-access-h7tjn\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.317222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.314855 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b783e0d4-8272-4c3e-b046-0d6a70160fea-istiod-ca-cert\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.317222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.314959 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-data\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.317222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.314999 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-credential-socket\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.416169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.416082 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-token\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.416169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.416143 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-workload-socket\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.416486 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.416221 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-workload-certs\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.416486 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.416269 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7tjn\" (UniqueName: \"kubernetes.io/projected/b783e0d4-8272-4c3e-b046-0d6a70160fea-kube-api-access-h7tjn\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.416486 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.416324 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b783e0d4-8272-4c3e-b046-0d6a70160fea-istiod-ca-cert\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.416486 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.416413 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-data\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.416486 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.416449 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-credential-socket\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.416667 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.416542 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-envoy\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.416667 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.416563 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-podinfo\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.416667 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.416595 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-workload-socket\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.416878 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.416857 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-credential-socket\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.416945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.416924 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-workload-certs\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.417448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.417423 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-data\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.417645 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.417464 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b783e0d4-8272-4c3e-b046-0d6a70160fea-istiod-ca-cert\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.419454 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.419426 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-envoy\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.419778 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.419754 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-podinfo\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.447088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.447015 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-token\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.448905 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.448879 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7tjn\" (UniqueName: \"kubernetes.io/projected/b783e0d4-8272-4c3e-b046-0d6a70160fea-kube-api-access-h7tjn\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.528528 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.528182 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:10.909245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.909166 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:12:10.946935 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.946850 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-istio-envoy\") pod \"f525a668-448c-44ed-8fac-21db37da4c29\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " Oct 29 23:12:10.947113 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.947016 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f525a668-448c-44ed-8fac-21db37da4c29-kube-api-access-lzzrk\" (UniqueName: \"kubernetes.io/projected/f525a668-448c-44ed-8fac-21db37da4c29-kube-api-access-lzzrk\") pod \"f525a668-448c-44ed-8fac-21db37da4c29\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " Oct 29 23:12:10.947113 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.947059 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f525a668-448c-44ed-8fac-21db37da4c29-istio-token\" (UniqueName: \"kubernetes.io/projected/f525a668-448c-44ed-8fac-21db37da4c29-istio-token\") pod \"f525a668-448c-44ed-8fac-21db37da4c29\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " Oct 29 23:12:10.947113 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.947086 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f525a668-448c-44ed-8fac-21db37da4c29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f525a668-448c-44ed-8fac-21db37da4c29-istio-podinfo\") pod \"f525a668-448c-44ed-8fac-21db37da4c29\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " Oct 29 23:12:10.947339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.947238 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-credential-socket\") pod \"f525a668-448c-44ed-8fac-21db37da4c29\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " Oct 29 23:12:10.947339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.947298 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-workload-certs\") pod \"f525a668-448c-44ed-8fac-21db37da4c29\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " Oct 29 23:12:10.947522 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.947443 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f525a668-448c-44ed-8fac-21db37da4c29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f525a668-448c-44ed-8fac-21db37da4c29-istiod-ca-cert\") pod \"f525a668-448c-44ed-8fac-21db37da4c29\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " Oct 29 23:12:10.947522 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.947499 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-istio-data\") pod \"f525a668-448c-44ed-8fac-21db37da4c29\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " Oct 29 23:12:10.947645 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.947529 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-credential-socket" pod "f525a668-448c-44ed-8fac-21db37da4c29" (UID: "f525a668-448c-44ed-8fac-21db37da4c29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:12:10.947645 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.947556 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-workload-socket\") pod \"f525a668-448c-44ed-8fac-21db37da4c29\" (UID: \"f525a668-448c-44ed-8fac-21db37da4c29\") " Oct 29 23:12:10.947864 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.947839 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f525a668-448c-44ed-8fac-21db37da4c29-istiod-ca-cert" pod "f525a668-448c-44ed-8fac-21db37da4c29" (UID: "f525a668-448c-44ed-8fac-21db37da4c29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:12:10.948163 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.948109 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-workload-socket" pod "f525a668-448c-44ed-8fac-21db37da4c29" (UID: "f525a668-448c-44ed-8fac-21db37da4c29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:12:10.948337 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.948316 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-istio-data" pod "f525a668-448c-44ed-8fac-21db37da4c29" (UID: "f525a668-448c-44ed-8fac-21db37da4c29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:12:10.948543 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.948518 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-workload-certs" pod "f525a668-448c-44ed-8fac-21db37da4c29" (UID: "f525a668-448c-44ed-8fac-21db37da4c29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:12:10.948633 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.948573 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:10.948633 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.948594 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f525a668-448c-44ed-8fac-21db37da4c29-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:10.948633 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.948608 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:10.948633 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.948620 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:10.949853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.949828 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f525a668-448c-44ed-8fac-21db37da4c29-kube-api-access-lzzrk" pod "f525a668-448c-44ed-8fac-21db37da4c29" (UID: "f525a668-448c-44ed-8fac-21db37da4c29"). InnerVolumeSpecName "kube-api-access-lzzrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:12:10.949977 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.949912 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f525a668-448c-44ed-8fac-21db37da4c29-istio-podinfo" pod "f525a668-448c-44ed-8fac-21db37da4c29" (UID: "f525a668-448c-44ed-8fac-21db37da4c29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:12:10.949977 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.949942 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f525a668-448c-44ed-8fac-21db37da4c29-istio-token" pod "f525a668-448c-44ed-8fac-21db37da4c29" (UID: "f525a668-448c-44ed-8fac-21db37da4c29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:12:10.950254 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.950236 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-istio-envoy" pod "f525a668-448c-44ed-8fac-21db37da4c29" (UID: "f525a668-448c-44ed-8fac-21db37da4c29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:12:10.962170 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.962142 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw"] Oct 29 23:12:10.963318 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:12:10.963287 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb783e0d4_8272_4c3e_b046_0d6a70160fea.slice/crio-b64ddaf5898463fb835a295d3914a0b0d58029a059c60f5109d4abf91df3346f WatchSource:0}: Error finding container b64ddaf5898463fb835a295d3914a0b0d58029a059c60f5109d4abf91df3346f: Status 404 returned error can't find the container with id b64ddaf5898463fb835a295d3914a0b0d58029a059c60f5109d4abf91df3346f Oct 29 23:12:10.966671 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.966636 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:12:10.966774 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.966712 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:12:10.966774 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:10.966744 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:12:11.049883 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.049849 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:11.049883 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.049877 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f525a668-448c-44ed-8fac-21db37da4c29-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:11.050062 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.049888 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzzrk\" (UniqueName: \"kubernetes.io/projected/f525a668-448c-44ed-8fac-21db37da4c29-kube-api-access-lzzrk\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:11.050062 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.049901 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f525a668-448c-44ed-8fac-21db37da4c29-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:11.050062 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.049910 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f525a668-448c-44ed-8fac-21db37da4c29-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:11.230052 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.229897 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" event={"ID":"b783e0d4-8272-4c3e-b046-0d6a70160fea","Type":"ContainerStarted","Data":"a9df19ba5de2e880e94d51add5dbaffa926cb5932164ed84a8b74f4607ad03d0"} Oct 29 23:12:11.230052 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.229981 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" event={"ID":"b783e0d4-8272-4c3e-b046-0d6a70160fea","Type":"ContainerStarted","Data":"b64ddaf5898463fb835a295d3914a0b0d58029a059c60f5109d4abf91df3346f"} Oct 29 23:12:11.232816 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.232791 5057 generic.go:356] "Generic (PLEG): container finished" podID="f525a668-448c-44ed-8fac-21db37da4c29" containerID="d9cb9b5f8f27dafff60b70ac0c64a0f3ff6446c7ba21c566e0ffe1d178fa4d18" exitCode=0 Oct 29 23:12:11.232964 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.232850 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" event={"ID":"f525a668-448c-44ed-8fac-21db37da4c29","Type":"ContainerDied","Data":"d9cb9b5f8f27dafff60b70ac0c64a0f3ff6446c7ba21c566e0ffe1d178fa4d18"} Oct 29 23:12:11.232964 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.232882 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" event={"ID":"f525a668-448c-44ed-8fac-21db37da4c29","Type":"ContainerDied","Data":"e5c04b3689e1fd7921138ea678c9d04e47da8e494b41f7cea6e8b884f8d118a6"} Oct 29 23:12:11.232964 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.232907 5057 scope.go:122] "RemoveContainer" containerID="d9cb9b5f8f27dafff60b70ac0c64a0f3ff6446c7ba21c566e0ffe1d178fa4d18" Oct 29 23:12:11.233126 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.233023 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx" Oct 29 23:12:11.256562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.256532 5057 scope.go:122] "RemoveContainer" containerID="3a15c28171201f3e2e413b776c629c6caeccc5bd03f816270cf1b4fb9e0f80e0" Oct 29 23:12:11.270393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.270274 5057 scope.go:122] "RemoveContainer" containerID="d9cb9b5f8f27dafff60b70ac0c64a0f3ff6446c7ba21c566e0ffe1d178fa4d18" Oct 29 23:12:11.270706 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:12:11.270674 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9cb9b5f8f27dafff60b70ac0c64a0f3ff6446c7ba21c566e0ffe1d178fa4d18\": container with ID starting with d9cb9b5f8f27dafff60b70ac0c64a0f3ff6446c7ba21c566e0ffe1d178fa4d18 not found: ID does not exist" containerID="d9cb9b5f8f27dafff60b70ac0c64a0f3ff6446c7ba21c566e0ffe1d178fa4d18" Oct 29 23:12:11.270799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.270718 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9cb9b5f8f27dafff60b70ac0c64a0f3ff6446c7ba21c566e0ffe1d178fa4d18"} err="failed to get container status \"d9cb9b5f8f27dafff60b70ac0c64a0f3ff6446c7ba21c566e0ffe1d178fa4d18\": rpc error: code = NotFound desc = could not find container \"d9cb9b5f8f27dafff60b70ac0c64a0f3ff6446c7ba21c566e0ffe1d178fa4d18\": container with ID starting with d9cb9b5f8f27dafff60b70ac0c64a0f3ff6446c7ba21c566e0ffe1d178fa4d18 not found: ID does not exist" Oct 29 23:12:11.270799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.270746 5057 scope.go:122] "RemoveContainer" containerID="3a15c28171201f3e2e413b776c629c6caeccc5bd03f816270cf1b4fb9e0f80e0" Oct 29 23:12:11.271079 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:12:11.271054 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a15c28171201f3e2e413b776c629c6caeccc5bd03f816270cf1b4fb9e0f80e0\": container with ID starting with 3a15c28171201f3e2e413b776c629c6caeccc5bd03f816270cf1b4fb9e0f80e0 not found: ID does not exist" containerID="3a15c28171201f3e2e413b776c629c6caeccc5bd03f816270cf1b4fb9e0f80e0" Oct 29 23:12:11.271136 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.271091 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a15c28171201f3e2e413b776c629c6caeccc5bd03f816270cf1b4fb9e0f80e0"} err="failed to get container status \"3a15c28171201f3e2e413b776c629c6caeccc5bd03f816270cf1b4fb9e0f80e0\": rpc error: code = NotFound desc = could not find container \"3a15c28171201f3e2e413b776c629c6caeccc5bd03f816270cf1b4fb9e0f80e0\": container with ID starting with 3a15c28171201f3e2e413b776c629c6caeccc5bd03f816270cf1b4fb9e0f80e0 not found: ID does not exist" Oct 29 23:12:11.288412 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.288351 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" podStartSLOduration=2.288334721 podStartE2EDuration="2.288334721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:11.282330499 +0000 UTC m=+1777.933829026" watchObservedRunningTime="2026-10-29 23:12:11.288334721 +0000 UTC m=+1777.939833256" Oct 29 23:12:11.338304 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.338270 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx"] Oct 29 23:12:11.338984 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.338962 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-phxbx"] Oct 29 23:12:11.534261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.533485 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f525a668-448c-44ed-8fac-21db37da4c29" path="/var/lib/kubelet/pods/f525a668-448c-44ed-8fac-21db37da4c29/volumes" Oct 29 23:12:11.534261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:11.533987 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:12.537103 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:12.536450 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:13.256308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:13.256270 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:13.257422 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:13.257397 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:12:18.070824 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:18.068403 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf"] Oct 29 23:12:18.070824 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:18.068643 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" podUID="82365a6f-eb57-45d8-bcd4-ff4e8daddf14" containerName="istio-proxy" containerID="cri-o://ed6cc162e1ddbd26cdb05d91cdd09f96533c2a1adc6d3821a53539fc053a32a0" gracePeriod=30 Oct 29 23:12:20.881646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:20.870803 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2"] Oct 29 23:12:20.881646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:20.871916 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f525a668-448c-44ed-8fac-21db37da4c29" containerName="istio-proxy" Oct 29 23:12:20.881646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:20.871961 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f525a668-448c-44ed-8fac-21db37da4c29" containerName="istio-proxy" Oct 29 23:12:20.881646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:20.871998 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f525a668-448c-44ed-8fac-21db37da4c29" containerName="istio-proxy" Oct 29 23:12:20.881646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:20.872006 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f525a668-448c-44ed-8fac-21db37da4c29" containerName="istio-proxy" Oct 29 23:12:20.881646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:20.872275 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f525a668-448c-44ed-8fac-21db37da4c29" containerName="istio-proxy" Oct 29 23:12:20.881646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:20.878013 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:20.900428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:20.900324 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-b4b5e3-openshift-default-dockercfg-td4sl\"" Oct 29 23:12:20.919634 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:20.919592 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2"] Oct 29 23:12:20.955745 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:20.955704 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-credential-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:20.955955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:20.955849 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-workload-certs\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:20.955955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:20.955909 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-workload-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:20.955955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:20.955934 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-token\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:20.956132 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:20.955967 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-envoy\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:20.956132 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:20.956073 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs7pf\" (UniqueName: \"kubernetes.io/projected/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-kube-api-access-hs7pf\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:20.956343 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:20.956148 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istiod-ca-cert\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:20.956343 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:20.956173 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-podinfo\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:20.956476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:20.956348 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-data\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:21.078289 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.060038 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-credential-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:21.078691 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.069120 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-credential-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:21.078961 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.078934 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-workload-certs\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:21.079148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.079119 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-workload-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:21.079295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.079280 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-token\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:21.079439 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.079425 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-envoy\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:21.079700 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.079682 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs7pf\" (UniqueName: \"kubernetes.io/projected/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-kube-api-access-hs7pf\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:21.079913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.079897 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istiod-ca-cert\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:21.093210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.080030 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-podinfo\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:21.093693 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.093657 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-data\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:21.094392 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.084110 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-workload-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:21.094565 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.083880 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-workload-certs\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:21.094683 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.094159 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-data\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:21.111438 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.098903 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istiod-ca-cert\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:21.117214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.116281 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-podinfo\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:21.146261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.145285 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-token\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:21.146261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.145733 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-envoy\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:21.160742 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.146816 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs7pf\" (UniqueName: \"kubernetes.io/projected/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-kube-api-access-hs7pf\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:21.199646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.198470 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:21.763275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.763247 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2"] Oct 29 23:12:21.763807 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:12:21.763779 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefd6fcf_b63a_4d1f_8b0a_8a3e6a033e31.slice/crio-6d8e351c2cb463c0b0aa963db95eabd89ae05763c63563d9835200af1b1f29d2 WatchSource:0}: Error finding container 6d8e351c2cb463c0b0aa963db95eabd89ae05763c63563d9835200af1b1f29d2: Status 404 returned error can't find the container with id 6d8e351c2cb463c0b0aa963db95eabd89ae05763c63563d9835200af1b1f29d2 Oct 29 23:12:21.766857 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.766823 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:12:21.766958 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.766889 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:12:21.766958 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:21.766920 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:12:22.475655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.475614 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" event={"ID":"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31","Type":"ContainerStarted","Data":"8329e3fc2f1ef0936c81518e85af99bc6108bb21fca5b0a93f0457bca99f0dda"} Oct 29 23:12:22.476128 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.475676 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" event={"ID":"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31","Type":"ContainerStarted","Data":"6d8e351c2cb463c0b0aa963db95eabd89ae05763c63563d9835200af1b1f29d2"} Oct 29 23:12:22.494229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.494171 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw"] Oct 29 23:12:22.495018 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.495001 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f525a668-448c-44ed-8fac-21db37da4c29" containerName="istio-proxy" Oct 29 23:12:22.498639 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.498622 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.522336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.522308 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp"] Oct 29 23:12:22.528178 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.527970 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw"] Oct 29 23:12:22.528178 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.528106 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.562803 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.560248 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" podStartSLOduration=2.560229812 podStartE2EDuration="2.560229812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:22.54652567 +0000 UTC m=+1789.198024223" watchObservedRunningTime="2026-10-29 23:12:22.560229812 +0000 UTC m=+1789.211728344" Oct 29 23:12:22.562803 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.560817 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp"] Oct 29 23:12:22.634847 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.634802 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.635031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.634868 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmv6j\" (UniqueName: \"kubernetes.io/projected/59a46795-a139-40f9-9eea-0a4f93fb244f-kube-api-access-kmv6j\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.635031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.634962 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.635031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.634996 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.635263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.635221 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.635326 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.635297 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.636342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.635500 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59a46795-a139-40f9-9eea-0a4f93fb244f-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.636342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.635547 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.636342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.635738 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.636342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.635775 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.636342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.635831 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.636342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.635870 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.636342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.635900 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.636342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.635953 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.636342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.636280 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.636342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.636312 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.637059 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.636442 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.637059 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.636503 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdhkc\" (UniqueName: \"kubernetes.io/projected/2ab845c5-deac-48c4-8a7c-b2dbada5d762-kube-api-access-zdhkc\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.740840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.738699 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.740840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.738793 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59a46795-a139-40f9-9eea-0a4f93fb244f-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.740840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.738822 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.740840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.738909 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.740840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.738945 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.740840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.738986 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.740840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.739025 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.740840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.739052 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.740840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.739088 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.740840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.739127 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.740840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.739154 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.740840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.739238 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.740840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.739319 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdhkc\" (UniqueName: \"kubernetes.io/projected/2ab845c5-deac-48c4-8a7c-b2dbada5d762-kube-api-access-zdhkc\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.740840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.739378 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.740840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.739424 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmv6j\" (UniqueName: \"kubernetes.io/projected/59a46795-a139-40f9-9eea-0a4f93fb244f-kube-api-access-kmv6j\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.740840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.739462 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.740840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.739494 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.743546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.739591 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.743546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.742423 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59a46795-a139-40f9-9eea-0a4f93fb244f-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.743546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.742552 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.743546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.742635 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.743546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.742905 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.743546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.743218 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.744133 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.743780 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.744481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.744451 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.744747 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.744722 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.744995 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.744974 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.752983 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.752955 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.759256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.758036 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.759715 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.759687 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.768799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.768775 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmv6j\" (UniqueName: \"kubernetes.io/projected/59a46795-a139-40f9-9eea-0a4f93fb244f-kube-api-access-kmv6j\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.769241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.769114 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.769523 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.769502 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.770016 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.769996 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:22.770128 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.770105 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.778668 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.778641 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdhkc\" (UniqueName: \"kubernetes.io/projected/2ab845c5-deac-48c4-8a7c-b2dbada5d762-kube-api-access-zdhkc\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.814741 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.814705 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:22.873879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:22.873499 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:23.179832 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.179799 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw"] Oct 29 23:12:23.199670 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.199580 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:23.205416 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.205391 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:23.220350 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:12:23.220315 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab845c5_deac_48c4_8a7c_b2dbada5d762.slice/crio-8782105af1db5d00ec5cddbf03c725cbf4a1664678aba385a09788b776132740 WatchSource:0}: Error finding container 8782105af1db5d00ec5cddbf03c725cbf4a1664678aba385a09788b776132740: Status 404 returned error can't find the container with id 8782105af1db5d00ec5cddbf03c725cbf4a1664678aba385a09788b776132740 Oct 29 23:12:23.224383 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.224347 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:12:23.224677 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.224454 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:12:23.224677 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.224498 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:12:23.230281 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:12:23.230246 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a46795_a139_40f9_9eea_0a4f93fb244f.slice/crio-b38ff1e6fd4bfc1b42aca531ec23326d35255175b516a2294003a2d64436839a WatchSource:0}: Error finding container b38ff1e6fd4bfc1b42aca531ec23326d35255175b516a2294003a2d64436839a: Status 404 returned error can't find the container with id b38ff1e6fd4bfc1b42aca531ec23326d35255175b516a2294003a2d64436839a Oct 29 23:12:23.234534 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.234514 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp"] Oct 29 23:12:23.236404 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.236373 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:12:23.236509 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.236463 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:12:23.236572 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.236512 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:12:23.402413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.402384 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:12:23.488957 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.488926 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-data\") pod \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " Oct 29 23:12:23.489523 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.489503 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istiod-ca-cert\") pod \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " Oct 29 23:12:23.489675 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.489660 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-workload-socket\") pod \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " Oct 29 23:12:23.489900 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.489888 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-workload-certs\") pod \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " Oct 29 23:12:23.490058 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.490045 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-envoy\") pod \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " Oct 29 23:12:23.490175 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.490163 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-kube-api-access-dttcp\" (UniqueName: \"kubernetes.io/projected/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-kube-api-access-dttcp\") pod \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " Oct 29 23:12:23.490359 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.490345 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-token\" (UniqueName: \"kubernetes.io/projected/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-token\") pod \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " Oct 29 23:12:23.490467 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.490455 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-podinfo\") pod \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " Oct 29 23:12:23.490640 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.490627 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-credential-socket\") pod \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\" (UID: \"82365a6f-eb57-45d8-bcd4-ff4e8daddf14\") " Oct 29 23:12:23.491661 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.491640 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-credential-socket" pod "82365a6f-eb57-45d8-bcd4-ff4e8daddf14" (UID: "82365a6f-eb57-45d8-bcd4-ff4e8daddf14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:12:23.492217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.491964 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istiod-ca-cert" pod "82365a6f-eb57-45d8-bcd4-ff4e8daddf14" (UID: "82365a6f-eb57-45d8-bcd4-ff4e8daddf14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:12:23.492638 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.492374 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-data" pod "82365a6f-eb57-45d8-bcd4-ff4e8daddf14" (UID: "82365a6f-eb57-45d8-bcd4-ff4e8daddf14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:12:23.493222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.493176 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-workload-socket" pod "82365a6f-eb57-45d8-bcd4-ff4e8daddf14" (UID: "82365a6f-eb57-45d8-bcd4-ff4e8daddf14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:12:23.493705 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.493684 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-workload-certs" pod "82365a6f-eb57-45d8-bcd4-ff4e8daddf14" (UID: "82365a6f-eb57-45d8-bcd4-ff4e8daddf14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:12:23.504529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.504481 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-kube-api-access-dttcp" pod "82365a6f-eb57-45d8-bcd4-ff4e8daddf14" (UID: "82365a6f-eb57-45d8-bcd4-ff4e8daddf14"). InnerVolumeSpecName "kube-api-access-dttcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:12:23.505035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.505012 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-podinfo" pod "82365a6f-eb57-45d8-bcd4-ff4e8daddf14" (UID: "82365a6f-eb57-45d8-bcd4-ff4e8daddf14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:12:23.515076 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.515035 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" event={"ID":"2ab845c5-deac-48c4-8a7c-b2dbada5d762","Type":"ContainerStarted","Data":"a73459a04103a5b698341b633722ca5fa92966f8dc1f720304188a58e5c08e6e"} Oct 29 23:12:23.515231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.515115 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" event={"ID":"2ab845c5-deac-48c4-8a7c-b2dbada5d762","Type":"ContainerStarted","Data":"8782105af1db5d00ec5cddbf03c725cbf4a1664678aba385a09788b776132740"} Oct 29 23:12:23.517276 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.517171 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-envoy" pod "82365a6f-eb57-45d8-bcd4-ff4e8daddf14" (UID: "82365a6f-eb57-45d8-bcd4-ff4e8daddf14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:12:23.518659 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.518628 5057 generic.go:356] "Generic (PLEG): container finished" podID="82365a6f-eb57-45d8-bcd4-ff4e8daddf14" containerID="ed6cc162e1ddbd26cdb05d91cdd09f96533c2a1adc6d3821a53539fc053a32a0" exitCode=0 Oct 29 23:12:23.518796 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.518774 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" event={"ID":"82365a6f-eb57-45d8-bcd4-ff4e8daddf14","Type":"ContainerDied","Data":"ed6cc162e1ddbd26cdb05d91cdd09f96533c2a1adc6d3821a53539fc053a32a0"} Oct 29 23:12:23.518880 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.518826 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" event={"ID":"82365a6f-eb57-45d8-bcd4-ff4e8daddf14","Type":"ContainerDied","Data":"296b52db72d8b014bab1c76b7d08a6543ee307e34f5293e60dbd8368b41f1dfc"} Oct 29 23:12:23.518880 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.518856 5057 scope.go:122] "RemoveContainer" containerID="ed6cc162e1ddbd26cdb05d91cdd09f96533c2a1adc6d3821a53539fc053a32a0" Oct 29 23:12:23.519025 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.519008 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf" Oct 29 23:12:23.522495 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.522199 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" event={"ID":"59a46795-a139-40f9-9eea-0a4f93fb244f","Type":"ContainerStarted","Data":"0650927cf63cfd6bd31ad94bfd71b41e23b3059dbe6b629e6d2618fe6259c932"} Oct 29 23:12:23.522495 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.522249 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" event={"ID":"59a46795-a139-40f9-9eea-0a4f93fb244f","Type":"ContainerStarted","Data":"b38ff1e6fd4bfc1b42aca531ec23326d35255175b516a2294003a2d64436839a"} Oct 29 23:12:23.522495 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.522280 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:23.523565 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.523487 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:12:23.529716 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.529649 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-token" pod "82365a6f-eb57-45d8-bcd4-ff4e8daddf14" (UID: "82365a6f-eb57-45d8-bcd4-ff4e8daddf14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:12:23.531237 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.531216 5057 scope.go:122] "RemoveContainer" containerID="760b92627049a7308e812ccd98760a670aeb09ff04746396f39b74ca578a090f" Oct 29 23:12:23.542736 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.542713 5057 scope.go:122] "RemoveContainer" containerID="ed6cc162e1ddbd26cdb05d91cdd09f96533c2a1adc6d3821a53539fc053a32a0" Oct 29 23:12:23.546344 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:12:23.546312 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed6cc162e1ddbd26cdb05d91cdd09f96533c2a1adc6d3821a53539fc053a32a0\": container with ID starting with ed6cc162e1ddbd26cdb05d91cdd09f96533c2a1adc6d3821a53539fc053a32a0 not found: ID does not exist" containerID="ed6cc162e1ddbd26cdb05d91cdd09f96533c2a1adc6d3821a53539fc053a32a0" Oct 29 23:12:23.546444 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.546354 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed6cc162e1ddbd26cdb05d91cdd09f96533c2a1adc6d3821a53539fc053a32a0"} err="failed to get container status \"ed6cc162e1ddbd26cdb05d91cdd09f96533c2a1adc6d3821a53539fc053a32a0\": rpc error: code = NotFound desc = could not find container \"ed6cc162e1ddbd26cdb05d91cdd09f96533c2a1adc6d3821a53539fc053a32a0\": container with ID starting with ed6cc162e1ddbd26cdb05d91cdd09f96533c2a1adc6d3821a53539fc053a32a0 not found: ID does not exist" Oct 29 23:12:23.546444 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.546379 5057 scope.go:122] "RemoveContainer" containerID="760b92627049a7308e812ccd98760a670aeb09ff04746396f39b74ca578a090f" Oct 29 23:12:23.546728 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:12:23.546704 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"760b92627049a7308e812ccd98760a670aeb09ff04746396f39b74ca578a090f\": container with ID starting with 760b92627049a7308e812ccd98760a670aeb09ff04746396f39b74ca578a090f not found: ID does not exist" containerID="760b92627049a7308e812ccd98760a670aeb09ff04746396f39b74ca578a090f" Oct 29 23:12:23.546813 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.546739 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"760b92627049a7308e812ccd98760a670aeb09ff04746396f39b74ca578a090f"} err="failed to get container status \"760b92627049a7308e812ccd98760a670aeb09ff04746396f39b74ca578a090f\": rpc error: code = NotFound desc = could not find container \"760b92627049a7308e812ccd98760a670aeb09ff04746396f39b74ca578a090f\": container with ID starting with 760b92627049a7308e812ccd98760a670aeb09ff04746396f39b74ca578a090f not found: ID does not exist" Oct 29 23:12:23.569685 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.569628 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" podStartSLOduration=1.5696090649999999 podStartE2EDuration="1.569609065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:23.564304246 +0000 UTC m=+1790.215802776" watchObservedRunningTime="2026-10-29 23:12:23.569609065 +0000 UTC m=+1790.221107596" Oct 29 23:12:23.593229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.593151 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:23.593229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.593217 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:23.593229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.593233 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dttcp\" (UniqueName: \"kubernetes.io/projected/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-kube-api-access-dttcp\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:23.593565 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.593272 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:23.593565 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.593287 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:23.593565 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.593298 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:23.593565 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.593310 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:23.593565 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.593323 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:23.593565 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.593334 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82365a6f-eb57-45d8-bcd4-ff4e8daddf14-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:23.689796 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.689732 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" podStartSLOduration=1.689712717 podStartE2EDuration="1.689712717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:23.681691241 +0000 UTC m=+1790.333189785" watchObservedRunningTime="2026-10-29 23:12:23.689712717 +0000 UTC m=+1790.341211257" Oct 29 23:12:23.814915 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.814875 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:23.874474 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.874414 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:23.883068 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.883032 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf"] Oct 29 23:12:23.887956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:23.887928 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-f2c8d3-openshift-default-995bfbf7c-rdqgf"] Oct 29 23:12:24.816782 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:24.816745 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 29 23:12:24.817240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:24.816798 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" podUID="2ab845c5-deac-48c4-8a7c-b2dbada5d762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 29 23:12:24.886845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:24.883898 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 29 23:12:24.886845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:24.883960 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" podUID="59a46795-a139-40f9-9eea-0a4f93fb244f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 29 23:12:25.687114 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:25.687079 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82365a6f-eb57-45d8-bcd4-ff4e8daddf14" path="/var/lib/kubelet/pods/82365a6f-eb57-45d8-bcd4-ff4e8daddf14/volumes" Oct 29 23:12:25.816000 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:25.815964 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 29 23:12:25.816218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:25.816026 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" podUID="2ab845c5-deac-48c4-8a7c-b2dbada5d762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 29 23:12:25.874781 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:25.874746 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 29 23:12:25.875406 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:25.875360 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" podUID="59a46795-a139-40f9-9eea-0a4f93fb244f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 29 23:12:26.815602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:26.815574 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 29 23:12:26.815837 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:26.815815 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" podUID="2ab845c5-deac-48c4-8a7c-b2dbada5d762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 29 23:12:26.874880 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:26.874845 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 29 23:12:26.885217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:26.883370 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" podUID="59a46795-a139-40f9-9eea-0a4f93fb244f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 29 23:12:27.815198 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:27.815152 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 29 23:12:27.815378 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:27.815248 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" podUID="2ab845c5-deac-48c4-8a7c-b2dbada5d762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 29 23:12:27.880207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:27.879700 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 29 23:12:27.880207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:27.879765 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" podUID="59a46795-a139-40f9-9eea-0a4f93fb244f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 29 23:12:28.815145 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:28.815103 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 29 23:12:28.815383 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:28.815164 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" podUID="2ab845c5-deac-48c4-8a7c-b2dbada5d762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 29 23:12:28.892286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:28.881444 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 29 23:12:28.892286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:28.881505 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" podUID="59a46795-a139-40f9-9eea-0a4f93fb244f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 29 23:12:29.817684 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:29.817645 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 29 23:12:29.818046 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:29.818001 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" podUID="2ab845c5-deac-48c4-8a7c-b2dbada5d762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 29 23:12:29.875604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:29.875565 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 29 23:12:29.875893 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:29.875864 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" podUID="59a46795-a139-40f9-9eea-0a4f93fb244f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 29 23:12:30.577143 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.577097 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc"] Oct 29 23:12:30.578079 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.578055 5057 cpu_manager.go:414] "containerMap: removing container" podUID="82365a6f-eb57-45d8-bcd4-ff4e8daddf14" containerName="istio-proxy" Oct 29 23:12:30.578079 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.578079 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82365a6f-eb57-45d8-bcd4-ff4e8daddf14" containerName="istio-proxy" Oct 29 23:12:30.578346 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.578330 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="82365a6f-eb57-45d8-bcd4-ff4e8daddf14" containerName="istio-proxy" Oct 29 23:12:30.578411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.578365 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="82365a6f-eb57-45d8-bcd4-ff4e8daddf14" containerName="istio-proxy" Oct 29 23:12:30.582941 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.582921 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" Oct 29 23:12:30.606479 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.606448 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc"] Oct 29 23:12:30.628230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.613867 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-26a2a4-serving-cert\"" Oct 29 23:12:30.650508 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.647670 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d1794a4-da94-40aa-80a4-9d62d8a809ca-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc\" (UID: \"5d1794a4-da94-40aa-80a4-9d62d8a809ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" Oct 29 23:12:30.650508 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.647715 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1794a4-da94-40aa-80a4-9d62d8a809ca-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc\" (UID: \"5d1794a4-da94-40aa-80a4-9d62d8a809ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" Oct 29 23:12:30.650508 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.647800 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8rmr\" (UniqueName: \"kubernetes.io/projected/5d1794a4-da94-40aa-80a4-9d62d8a809ca-kube-api-access-p8rmr\") pod \"maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc\" (UID: \"5d1794a4-da94-40aa-80a4-9d62d8a809ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" Oct 29 23:12:30.650508 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.647848 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d1794a4-da94-40aa-80a4-9d62d8a809ca-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc\" (UID: \"5d1794a4-da94-40aa-80a4-9d62d8a809ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" Oct 29 23:12:30.749243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.749202 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d1794a4-da94-40aa-80a4-9d62d8a809ca-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc\" (UID: \"5d1794a4-da94-40aa-80a4-9d62d8a809ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" Oct 29 23:12:30.749429 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.749264 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1794a4-da94-40aa-80a4-9d62d8a809ca-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc\" (UID: \"5d1794a4-da94-40aa-80a4-9d62d8a809ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" Oct 29 23:12:30.749429 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.749385 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8rmr\" (UniqueName: \"kubernetes.io/projected/5d1794a4-da94-40aa-80a4-9d62d8a809ca-kube-api-access-p8rmr\") pod \"maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc\" (UID: \"5d1794a4-da94-40aa-80a4-9d62d8a809ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" Oct 29 23:12:30.749511 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.749466 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d1794a4-da94-40aa-80a4-9d62d8a809ca-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc\" (UID: \"5d1794a4-da94-40aa-80a4-9d62d8a809ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" Oct 29 23:12:30.751239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.751207 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1794a4-da94-40aa-80a4-9d62d8a809ca-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc\" (UID: \"5d1794a4-da94-40aa-80a4-9d62d8a809ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" Oct 29 23:12:30.757750 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.757724 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d1794a4-da94-40aa-80a4-9d62d8a809ca-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc\" (UID: \"5d1794a4-da94-40aa-80a4-9d62d8a809ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" Oct 29 23:12:30.757882 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.757826 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d1794a4-da94-40aa-80a4-9d62d8a809ca-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc\" (UID: \"5d1794a4-da94-40aa-80a4-9d62d8a809ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" Oct 29 23:12:30.776919 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.776891 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8rmr\" (UniqueName: \"kubernetes.io/projected/5d1794a4-da94-40aa-80a4-9d62d8a809ca-kube-api-access-p8rmr\") pod \"maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc\" (UID: \"5d1794a4-da94-40aa-80a4-9d62d8a809ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" Oct 29 23:12:30.831960 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.823299 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 29 23:12:30.831960 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.823357 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" podUID="2ab845c5-deac-48c4-8a7c-b2dbada5d762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 29 23:12:30.880922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.877340 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 29 23:12:30.880922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.877451 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" podUID="59a46795-a139-40f9-9eea-0a4f93fb244f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 29 23:12:30.944313 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:30.907644 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" Oct 29 23:12:31.429234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:31.429181 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc"] Oct 29 23:12:31.429989 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:12:31.429954 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1794a4_da94_40aa_80a4_9d62d8a809ca.slice/crio-1a458933369bbd2e94d417fdf2b9b9ffadfc87e5bb35b5c73b7227a8aa16c4c2 WatchSource:0}: Error finding container 1a458933369bbd2e94d417fdf2b9b9ffadfc87e5bb35b5c73b7227a8aa16c4c2: Status 404 returned error can't find the container with id 1a458933369bbd2e94d417fdf2b9b9ffadfc87e5bb35b5c73b7227a8aa16c4c2 Oct 29 23:12:31.917474 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:31.914417 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" event={"ID":"5d1794a4-da94-40aa-80a4-9d62d8a809ca","Type":"ContainerStarted","Data":"1a458933369bbd2e94d417fdf2b9b9ffadfc87e5bb35b5c73b7227a8aa16c4c2"} Oct 29 23:12:32.539386 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:32.539227 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:32.539715 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:32.539680 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:32.542232 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:32.541284 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:12:32.623675 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:32.623643 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:32.623965 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:32.623942 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:32.625590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:32.625564 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:12:32.932307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:32.932268 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" event={"ID":"5d1794a4-da94-40aa-80a4-9d62d8a809ca","Type":"ContainerStarted","Data":"813038b0fd11954f359d9a2070b1363c75176b239b3fbc5dfb2cdedbfa835709"} Oct 29 23:12:32.933043 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:32.933018 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" Oct 29 23:12:32.981905 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:32.981847 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" podStartSLOduration=2.686265936 podStartE2EDuration="2.981831321s" totalImagesPullingTime="295.565385ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:32.97966285 +0000 UTC m=+1799.631161378" watchObservedRunningTime="2026-10-29 23:12:32.981831321 +0000 UTC m=+1799.633329863" Oct 29 23:12:33.914272 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:12:33.914236 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:12:33.970566 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:33.970534 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh"] Oct 29 23:12:33.971323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:33.971305 5057 cpu_manager.go:414] "containerMap: removing container" podUID="82365a6f-eb57-45d8-bcd4-ff4e8daddf14" containerName="istio-proxy" Oct 29 23:12:33.971323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:33.971323 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82365a6f-eb57-45d8-bcd4-ff4e8daddf14" containerName="istio-proxy" Oct 29 23:12:33.975425 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:33.975399 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.012338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.011894 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-2126fb-openshift-default-dockercfg-2swzx\"" Oct 29 23:12:34.016658 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.016632 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh"] Oct 29 23:12:34.054206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.053664 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-istio-data\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.054206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.053758 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-istio-envoy\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.054206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.053806 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-workload-certs\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.054206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.053832 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c287175-5866-4c13-b806-5839582839cc-istiod-ca-cert\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.054206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.053866 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c287175-5866-4c13-b806-5839582839cc-istio-token\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.054206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.053940 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c287175-5866-4c13-b806-5839582839cc-istio-podinfo\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.054206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.053996 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-workload-socket\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.054206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.054025 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwf7x\" (UniqueName: \"kubernetes.io/projected/5c287175-5866-4c13-b806-5839582839cc-kube-api-access-xwf7x\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.054206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.054071 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-credential-socket\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.356206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.308656 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-istio-data\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.356206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.308783 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-istio-envoy\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.356206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.308839 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-workload-certs\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.356206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.308875 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c287175-5866-4c13-b806-5839582839cc-istiod-ca-cert\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.356206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.309069 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c287175-5866-4c13-b806-5839582839cc-istio-token\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.356206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.324050 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c287175-5866-4c13-b806-5839582839cc-istio-podinfo\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.356206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.324223 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-workload-socket\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.356206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.324276 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwf7x\" (UniqueName: \"kubernetes.io/projected/5c287175-5866-4c13-b806-5839582839cc-kube-api-access-xwf7x\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.356206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.324368 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-credential-socket\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.356206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.325746 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-credential-socket\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.408245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.400917 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-istio-envoy\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.408245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.402032 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c287175-5866-4c13-b806-5839582839cc-istiod-ca-cert\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.453206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.433297 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-workload-socket\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.453206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.433664 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-istio-data\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.453206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.434269 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-workload-certs\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.453206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.443018 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c287175-5866-4c13-b806-5839582839cc-istio-podinfo\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.453206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.445518 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwf7x\" (UniqueName: \"kubernetes.io/projected/5c287175-5866-4c13-b806-5839582839cc-kube-api-access-xwf7x\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.541229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.541177 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c287175-5866-4c13-b806-5839582839cc-istio-token\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.611454 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.611370 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-2126fb-openshift-default-dockercfg-2swzx\"" Oct 29 23:12:34.616460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.616422 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:34.818271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.818239 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:12:34.861057 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:34.861026 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:12:35.044458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:35.043502 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh"] Oct 29 23:12:35.051957 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:12:35.051915 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c287175_5866_4c13_b806_5839582839cc.slice/crio-83e9057a3cbb91884a3e76fc04619aad6f8f6c8a33c6105bf682a57348f4a405 WatchSource:0}: Error finding container 83e9057a3cbb91884a3e76fc04619aad6f8f6c8a33c6105bf682a57348f4a405: Status 404 returned error can't find the container with id 83e9057a3cbb91884a3e76fc04619aad6f8f6c8a33c6105bf682a57348f4a405 Oct 29 23:12:35.059444 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:35.055413 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:12:35.059444 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:35.055501 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:12:35.059444 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:35.055549 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:12:35.988124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:35.988090 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" event={"ID":"5c287175-5866-4c13-b806-5839582839cc","Type":"ContainerStarted","Data":"081524ad795f0d79617a18403c6e36507788c2a4e9441e0832a1e3ccfbb6fbd0"} Oct 29 23:12:35.988326 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:35.988149 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" event={"ID":"5c287175-5866-4c13-b806-5839582839cc","Type":"ContainerStarted","Data":"83e9057a3cbb91884a3e76fc04619aad6f8f6c8a33c6105bf682a57348f4a405"} Oct 29 23:12:36.055013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:36.054947 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" podStartSLOduration=3.054928738 podStartE2EDuration="3.054928738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:36.051642007 +0000 UTC m=+1802.703140540" watchObservedRunningTime="2026-10-29 23:12:36.054928738 +0000 UTC m=+1802.706427271" Oct 29 23:12:36.617217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:36.617153 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:36.624237 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:36.624212 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:36.995540 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:36.995508 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:36.996560 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:36.996539 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:12:40.028215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:40.027366 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" Oct 29 23:12:40.554227 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:40.554160 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd"] Oct 29 23:12:40.554431 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:40.554398 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd" podUID="e5272969-d30b-4e39-93b3-587d000ef056" containerName="payload-processing" containerID="cri-o://04ce560694505d4f88ad0c756977e53fe43b4470ec8b240916b1637ae71f1944" gracePeriod=30 Oct 29 23:12:40.587439 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:40.587393 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl"] Oct 29 23:12:40.587771 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:40.587682 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl" podUID="6b843c0e-5b58-46ef-b5cd-712b4acad831" containerName="payload-pre-processing" containerID="cri-o://c65ff59275a4316f041040164c9b9e8a05ff36241be2f6acbcaac8aaea7a3ed1" gracePeriod=30 Oct 29 23:12:41.344257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:41.343981 5057 generic.go:356] "Generic (PLEG): container finished" podID="e5272969-d30b-4e39-93b3-587d000ef056" containerID="04ce560694505d4f88ad0c756977e53fe43b4470ec8b240916b1637ae71f1944" exitCode=0 Oct 29 23:12:41.344257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:41.344137 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd" event={"ID":"e5272969-d30b-4e39-93b3-587d000ef056","Type":"ContainerDied","Data":"04ce560694505d4f88ad0c756977e53fe43b4470ec8b240916b1637ae71f1944"} Oct 29 23:12:41.348180 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:41.347506 5057 generic.go:356] "Generic (PLEG): container finished" podID="6b843c0e-5b58-46ef-b5cd-712b4acad831" containerID="c65ff59275a4316f041040164c9b9e8a05ff36241be2f6acbcaac8aaea7a3ed1" exitCode=0 Oct 29 23:12:41.348180 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:41.347725 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl" event={"ID":"6b843c0e-5b58-46ef-b5cd-712b4acad831","Type":"ContainerDied","Data":"c65ff59275a4316f041040164c9b9e8a05ff36241be2f6acbcaac8aaea7a3ed1"} Oct 29 23:12:41.434789 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:41.434749 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl" Oct 29 23:12:41.440442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:41.440398 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd" Oct 29 23:12:41.578248 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:41.578177 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5272969-d30b-4e39-93b3-587d000ef056-kube-api-access-6pb2p\" (UniqueName: \"kubernetes.io/projected/e5272969-d30b-4e39-93b3-587d000ef056-kube-api-access-6pb2p\") pod \"e5272969-d30b-4e39-93b3-587d000ef056\" (UID: \"e5272969-d30b-4e39-93b3-587d000ef056\") " Oct 29 23:12:41.578449 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:41.578369 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b843c0e-5b58-46ef-b5cd-712b4acad831-kube-api-access-fh6wn\" (UniqueName: \"kubernetes.io/projected/6b843c0e-5b58-46ef-b5cd-712b4acad831-kube-api-access-fh6wn\") pod \"6b843c0e-5b58-46ef-b5cd-712b4acad831\" (UID: \"6b843c0e-5b58-46ef-b5cd-712b4acad831\") " Oct 29 23:12:41.578449 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:41.578417 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b843c0e-5b58-46ef-b5cd-712b4acad831-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b843c0e-5b58-46ef-b5cd-712b4acad831-plugins-config-volume\") pod \"6b843c0e-5b58-46ef-b5cd-712b4acad831\" (UID: \"6b843c0e-5b58-46ef-b5cd-712b4acad831\") " Oct 29 23:12:41.578569 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:41.578510 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5272969-d30b-4e39-93b3-587d000ef056-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5272969-d30b-4e39-93b3-587d000ef056-plugins-config-volume\") pod \"e5272969-d30b-4e39-93b3-587d000ef056\" (UID: \"e5272969-d30b-4e39-93b3-587d000ef056\") " Oct 29 23:12:41.583148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:41.583115 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5272969-d30b-4e39-93b3-587d000ef056-kube-api-access-6pb2p" pod "e5272969-d30b-4e39-93b3-587d000ef056" (UID: "e5272969-d30b-4e39-93b3-587d000ef056"). InnerVolumeSpecName "kube-api-access-6pb2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:12:41.584544 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:41.584524 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5272969-d30b-4e39-93b3-587d000ef056-plugins-config-volume" pod "e5272969-d30b-4e39-93b3-587d000ef056" (UID: "e5272969-d30b-4e39-93b3-587d000ef056"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:12:41.585044 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:41.585028 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b843c0e-5b58-46ef-b5cd-712b4acad831-plugins-config-volume" pod "6b843c0e-5b58-46ef-b5cd-712b4acad831" (UID: "6b843c0e-5b58-46ef-b5cd-712b4acad831"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:12:41.716212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:41.710452 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b843c0e-5b58-46ef-b5cd-712b4acad831-kube-api-access-fh6wn" pod "6b843c0e-5b58-46ef-b5cd-712b4acad831" (UID: "6b843c0e-5b58-46ef-b5cd-712b4acad831"). InnerVolumeSpecName "kube-api-access-fh6wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:12:41.716212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:41.711601 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pb2p\" (UniqueName: \"kubernetes.io/projected/e5272969-d30b-4e39-93b3-587d000ef056-kube-api-access-6pb2p\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:41.716212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:41.711635 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh6wn\" (UniqueName: \"kubernetes.io/projected/6b843c0e-5b58-46ef-b5cd-712b4acad831-kube-api-access-fh6wn\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:41.716212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:41.711650 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b843c0e-5b58-46ef-b5cd-712b4acad831-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:41.716212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:41.711665 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5272969-d30b-4e39-93b3-587d000ef056-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:42.465619 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:42.465582 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl" event={"ID":"6b843c0e-5b58-46ef-b5cd-712b4acad831","Type":"ContainerDied","Data":"33d8c947f1ed647b98c276a642d1bc5219a403fda45129d756af19a2026926bb"} Oct 29 23:12:42.466216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:42.466170 5057 scope.go:122] "RemoveContainer" containerID="c65ff59275a4316f041040164c9b9e8a05ff36241be2f6acbcaac8aaea7a3ed1" Oct 29 23:12:42.466605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:42.466588 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl" Oct 29 23:12:42.484797 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:42.484752 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd" event={"ID":"e5272969-d30b-4e39-93b3-587d000ef056","Type":"ContainerDied","Data":"a1765b3af47f9da3758ddf47f64a8208f37ae048bc871a5fc60fa377a0c830c3"} Oct 29 23:12:42.485129 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:42.485097 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd" Oct 29 23:12:42.686147 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:42.683882 5057 scope.go:122] "RemoveContainer" containerID="04ce560694505d4f88ad0c756977e53fe43b4470ec8b240916b1637ae71f1944" Oct 29 23:12:42.781516 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:42.781477 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd"] Oct 29 23:12:42.884370 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:42.884245 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-618d6c-84dcd46c64-4lpxd"] Oct 29 23:12:42.913025 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:42.912944 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl"] Oct 29 23:12:42.924697 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:42.921134 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-66c9c6d7bc-p52wl"] Oct 29 23:12:43.063260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:43.062742 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:12:43.142069 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:43.140331 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 23:12:43.177370 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:43.177323 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:43.717211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:43.714267 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:43.735903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:43.735868 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b843c0e-5b58-46ef-b5cd-712b4acad831" path="/var/lib/kubelet/pods/6b843c0e-5b58-46ef-b5cd-712b4acad831/volumes" Oct 29 23:12:43.736875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:43.736847 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5272969-d30b-4e39-93b3-587d000ef056" path="/var/lib/kubelet/pods/e5272969-d30b-4e39-93b3-587d000ef056/volumes" Oct 29 23:12:44.336210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:44.200262 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:44.336210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:44.248320 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:12:44.336210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:44.257789 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:12:44.336210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:44.261836 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 23:12:44.336210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:44.266488 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:12:44.336210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:44.266550 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:12:44.585133 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:44.585081 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded" Oct 29 23:12:44.730211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:44.726569 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:44.730211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:44.727531 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:12:45.232667 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:45.211268 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:45.254454 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:45.243303 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:45.264604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:45.264565 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:45.299276 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:45.273274 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:12:45.311225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:45.307011 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:12:45.311225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:45.307075 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" podUID="e794e7ae-93a6-4789-9542-217137d0aa1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:45.758755 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:45.758709 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 23:12:46.003213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:45.987672 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:12:46.003213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:45.987763 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" podUID="503f6577-e8f6-474e-8b57-98e30edaa785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:46.061270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:46.038179 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded" Oct 29 23:12:46.182210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:46.178401 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:12:46.182210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:46.178486 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" podUID="c5db1599-6906-4dcd-8182-787026e2e745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:46.218562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:46.218517 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:46.286585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:46.286530 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:46.340438 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:46.313794 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:12:46.770209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:46.770142 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:47.312837 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:47.312792 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:47.337001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:47.336884 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:47.338662 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:47.338625 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:47.394895 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:47.394862 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:12:47.447571 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:47.447540 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:12:47.458509 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:47.458475 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:12:47.458789 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:47.458759 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" podUID="7ea22168-c52a-43f3-a4b6-4cdd87cffdae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:47.470205 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:47.470158 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:12:47.708630 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:47.708599 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:12:49.117230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:49.117163 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs"] Oct 29 23:12:49.117750 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:49.117467 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" podUID="34c31c14-402a-4cff-a9e1-0a6940ffc05a" containerName="istio-proxy" containerID="cri-o://c5709440bc986a3e3743775769bfe6351e5d65476746ec9daeea3122b730a932" gracePeriod=30 Oct 29 23:12:54.568445 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.568410 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:12:54.676742 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.676695 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-credential-socket\") pod \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " Oct 29 23:12:54.676924 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.676772 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-token\" (UniqueName: \"kubernetes.io/projected/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-token\") pod \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " Oct 29 23:12:54.676924 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.676847 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-workload-socket\") pod \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " Oct 29 23:12:54.677069 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.676929 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-workload-certs\") pod \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " Oct 29 23:12:54.677069 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.677028 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-data\") pod \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " Oct 29 23:12:54.677069 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.677046 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-credential-socket" pod "34c31c14-402a-4cff-a9e1-0a6940ffc05a" (UID: "34c31c14-402a-4cff-a9e1-0a6940ffc05a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:12:54.677244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.677140 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-envoy\") pod \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " Oct 29 23:12:54.677244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.677223 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istiod-ca-cert\") pod \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " Oct 29 23:12:54.677359 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.677295 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-workload-socket" pod "34c31c14-402a-4cff-a9e1-0a6940ffc05a" (UID: "34c31c14-402a-4cff-a9e1-0a6940ffc05a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:12:54.677419 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.677393 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c31c14-402a-4cff-a9e1-0a6940ffc05a-kube-api-access-cxw9l\" (UniqueName: \"kubernetes.io/projected/34c31c14-402a-4cff-a9e1-0a6940ffc05a-kube-api-access-cxw9l\") pod \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " Oct 29 23:12:54.677517 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.677490 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-podinfo\") pod \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\" (UID: \"34c31c14-402a-4cff-a9e1-0a6940ffc05a\") " Oct 29 23:12:54.678087 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.678059 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-workload-certs" pod "34c31c14-402a-4cff-a9e1-0a6940ffc05a" (UID: "34c31c14-402a-4cff-a9e1-0a6940ffc05a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:12:54.678673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.678648 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-data" pod "34c31c14-402a-4cff-a9e1-0a6940ffc05a" (UID: "34c31c14-402a-4cff-a9e1-0a6940ffc05a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:12:54.679911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.679886 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istiod-ca-cert" pod "34c31c14-402a-4cff-a9e1-0a6940ffc05a" (UID: "34c31c14-402a-4cff-a9e1-0a6940ffc05a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:12:54.680403 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.680375 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:54.680403 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.680408 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:54.680403 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.680423 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:54.680403 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.680436 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:54.680721 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.680450 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:54.682328 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.682295 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c31c14-402a-4cff-a9e1-0a6940ffc05a-kube-api-access-cxw9l" pod "34c31c14-402a-4cff-a9e1-0a6940ffc05a" (UID: "34c31c14-402a-4cff-a9e1-0a6940ffc05a"). InnerVolumeSpecName "kube-api-access-cxw9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:12:54.684096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.684036 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-token" pod "34c31c14-402a-4cff-a9e1-0a6940ffc05a" (UID: "34c31c14-402a-4cff-a9e1-0a6940ffc05a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:12:54.684265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.684242 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-envoy" pod "34c31c14-402a-4cff-a9e1-0a6940ffc05a" (UID: "34c31c14-402a-4cff-a9e1-0a6940ffc05a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:12:54.684581 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.684556 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-podinfo" pod "34c31c14-402a-4cff-a9e1-0a6940ffc05a" (UID: "34c31c14-402a-4cff-a9e1-0a6940ffc05a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:12:54.782249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.782163 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:54.782249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.782210 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:54.782249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.782220 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxw9l\" (UniqueName: \"kubernetes.io/projected/34c31c14-402a-4cff-a9e1-0a6940ffc05a-kube-api-access-cxw9l\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:54.782249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:54.782231 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c31c14-402a-4cff-a9e1-0a6940ffc05a-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:12:55.033297 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:55.032760 5057 generic.go:356] "Generic (PLEG): container finished" podID="34c31c14-402a-4cff-a9e1-0a6940ffc05a" containerID="c5709440bc986a3e3743775769bfe6351e5d65476746ec9daeea3122b730a932" exitCode=0 Oct 29 23:12:55.033297 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:55.032873 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" event={"ID":"34c31c14-402a-4cff-a9e1-0a6940ffc05a","Type":"ContainerDied","Data":"c5709440bc986a3e3743775769bfe6351e5d65476746ec9daeea3122b730a932"} Oct 29 23:12:55.033297 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:55.032971 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" event={"ID":"34c31c14-402a-4cff-a9e1-0a6940ffc05a","Type":"ContainerDied","Data":"6690c478c526fef225607f7a442c096b98c6071e9b354b7517e1914483ecb265"} Oct 29 23:12:55.033297 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:55.033002 5057 scope.go:122] "RemoveContainer" containerID="c5709440bc986a3e3743775769bfe6351e5d65476746ec9daeea3122b730a932" Oct 29 23:12:55.033297 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:55.033207 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs" Oct 29 23:12:55.096524 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:55.096496 5057 scope.go:122] "RemoveContainer" containerID="b781f2aa745b0adda57e926bfbcc188a00e31148dfd3d50dba58ab4db389f26b" Oct 29 23:12:55.109697 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:55.109674 5057 scope.go:122] "RemoveContainer" containerID="c5709440bc986a3e3743775769bfe6351e5d65476746ec9daeea3122b730a932" Oct 29 23:12:55.110055 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:12:55.110031 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5709440bc986a3e3743775769bfe6351e5d65476746ec9daeea3122b730a932\": container with ID starting with c5709440bc986a3e3743775769bfe6351e5d65476746ec9daeea3122b730a932 not found: ID does not exist" containerID="c5709440bc986a3e3743775769bfe6351e5d65476746ec9daeea3122b730a932" Oct 29 23:12:55.110135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:55.110065 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5709440bc986a3e3743775769bfe6351e5d65476746ec9daeea3122b730a932"} err="failed to get container status \"c5709440bc986a3e3743775769bfe6351e5d65476746ec9daeea3122b730a932\": rpc error: code = NotFound desc = could not find container \"c5709440bc986a3e3743775769bfe6351e5d65476746ec9daeea3122b730a932\": container with ID starting with c5709440bc986a3e3743775769bfe6351e5d65476746ec9daeea3122b730a932 not found: ID does not exist" Oct 29 23:12:55.110135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:55.110085 5057 scope.go:122] "RemoveContainer" containerID="b781f2aa745b0adda57e926bfbcc188a00e31148dfd3d50dba58ab4db389f26b" Oct 29 23:12:55.110450 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:12:55.110427 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b781f2aa745b0adda57e926bfbcc188a00e31148dfd3d50dba58ab4db389f26b\": container with ID starting with b781f2aa745b0adda57e926bfbcc188a00e31148dfd3d50dba58ab4db389f26b not found: ID does not exist" containerID="b781f2aa745b0adda57e926bfbcc188a00e31148dfd3d50dba58ab4db389f26b" Oct 29 23:12:55.110511 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:55.110457 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b781f2aa745b0adda57e926bfbcc188a00e31148dfd3d50dba58ab4db389f26b"} err="failed to get container status \"b781f2aa745b0adda57e926bfbcc188a00e31148dfd3d50dba58ab4db389f26b\": rpc error: code = NotFound desc = could not find container \"b781f2aa745b0adda57e926bfbcc188a00e31148dfd3d50dba58ab4db389f26b\": container with ID starting with b781f2aa745b0adda57e926bfbcc188a00e31148dfd3d50dba58ab4db389f26b not found: ID does not exist" Oct 29 23:12:55.188464 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:55.188433 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs"] Oct 29 23:12:55.192050 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:55.192020 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-618d6c-openshift-default-8654d45985-ttqvs"] Oct 29 23:12:55.483432 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:55.483395 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34c31c14-402a-4cff-a9e1-0a6940ffc05a" path="/var/lib/kubelet/pods/34c31c14-402a-4cff-a9e1-0a6940ffc05a/volumes" Oct 29 23:12:56.379343 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.379312 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm"] Oct 29 23:12:56.380024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.380008 5057 cpu_manager.go:414] "containerMap: removing container" podUID="34c31c14-402a-4cff-a9e1-0a6940ffc05a" containerName="istio-proxy" Oct 29 23:12:56.380086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.380024 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34c31c14-402a-4cff-a9e1-0a6940ffc05a" containerName="istio-proxy" Oct 29 23:12:56.380086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.380047 5057 cpu_manager.go:414] "containerMap: removing container" podUID="34c31c14-402a-4cff-a9e1-0a6940ffc05a" containerName="istio-proxy" Oct 29 23:12:56.380086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.380053 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34c31c14-402a-4cff-a9e1-0a6940ffc05a" containerName="istio-proxy" Oct 29 23:12:56.380086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.380064 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e5272969-d30b-4e39-93b3-587d000ef056" containerName="payload-processing" Oct 29 23:12:56.380086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.380069 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5272969-d30b-4e39-93b3-587d000ef056" containerName="payload-processing" Oct 29 23:12:56.380282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.380099 5057 cpu_manager.go:414] "containerMap: removing container" podUID="6b843c0e-5b58-46ef-b5cd-712b4acad831" containerName="payload-pre-processing" Oct 29 23:12:56.380282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.380104 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b843c0e-5b58-46ef-b5cd-712b4acad831" containerName="payload-pre-processing" Oct 29 23:12:56.380282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.380220 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="34c31c14-402a-4cff-a9e1-0a6940ffc05a" containerName="istio-proxy" Oct 29 23:12:56.380282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.380229 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5272969-d30b-4e39-93b3-587d000ef056" containerName="payload-processing" Oct 29 23:12:56.380282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.380241 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b843c0e-5b58-46ef-b5cd-712b4acad831" containerName="payload-pre-processing" Oct 29 23:12:56.385361 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.385339 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.406674 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.406037 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-7a3c1a-openshift-default-dockercfg-69xwl\"" Oct 29 23:12:56.435347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.435311 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm"] Oct 29 23:12:56.531209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.530472 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hflnr\" (UniqueName: \"kubernetes.io/projected/fa2b610a-d82f-47f8-93cd-736a51e1d52e-kube-api-access-hflnr\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.531209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.530534 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-workload-certs\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.531209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.530567 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-data\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.531209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.530806 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-envoy\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.531209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.530893 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-workload-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.531209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.531012 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istiod-ca-cert\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.531209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.531057 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-token\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.531209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.531084 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-credential-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.531209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.531109 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-podinfo\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.643270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.635270 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-credential-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.643270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.635368 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-podinfo\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.643270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.635456 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hflnr\" (UniqueName: \"kubernetes.io/projected/fa2b610a-d82f-47f8-93cd-736a51e1d52e-kube-api-access-hflnr\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.643270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.635495 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-workload-certs\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.643270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.635520 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-data\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.643270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.635728 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-envoy\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.643270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.635847 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-workload-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.643270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.636038 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istiod-ca-cert\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.643270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.636095 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-token\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.643270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.636966 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-credential-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.695161 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.695122 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-workload-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.695710 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.695684 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-data\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.696124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.696101 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-workload-certs\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.697493 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.697471 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istiod-ca-cert\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.718276 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.718104 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hflnr\" (UniqueName: \"kubernetes.io/projected/fa2b610a-d82f-47f8-93cd-736a51e1d52e-kube-api-access-hflnr\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.786859 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.786763 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-token\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.788129 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.787968 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-podinfo\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:56.819873 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:56.819841 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-envoy\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:57.038080 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:57.038049 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:57.673790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:57.673633 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm"] Oct 29 23:12:57.703733 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:12:57.703689 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2b610a_d82f_47f8_93cd_736a51e1d52e.slice/crio-0dabd74b92673ac2d89e358fb49b120b8eb3d99d60ecfe11f149336244411b33 WatchSource:0}: Error finding container 0dabd74b92673ac2d89e358fb49b120b8eb3d99d60ecfe11f149336244411b33: Status 404 returned error can't find the container with id 0dabd74b92673ac2d89e358fb49b120b8eb3d99d60ecfe11f149336244411b33 Oct 29 23:12:57.707666 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:57.707616 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:12:57.707795 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:57.707720 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:12:57.707795 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:57.707767 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:12:58.181623 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:58.180693 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" event={"ID":"fa2b610a-d82f-47f8-93cd-736a51e1d52e","Type":"ContainerStarted","Data":"7f391a6bddc7cb5aa23ac88ae5dfa48dfdee05eb3e2f6a74adeac80c89a97845"} Oct 29 23:12:58.181623 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:58.180829 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" event={"ID":"fa2b610a-d82f-47f8-93cd-736a51e1d52e","Type":"ContainerStarted","Data":"0dabd74b92673ac2d89e358fb49b120b8eb3d99d60ecfe11f149336244411b33"} Oct 29 23:12:58.310215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:58.300461 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" podStartSLOduration=2.300439576 podStartE2EDuration="2.300439576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:58.258938227 +0000 UTC m=+1824.910436758" watchObservedRunningTime="2026-10-29 23:12:58.300439576 +0000 UTC m=+1824.951938107" Oct 29 23:12:59.038274 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:59.038229 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:59.047292 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:59.047260 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:59.190533 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:59.190499 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:12:59.191596 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:12:59.191571 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:13:01.885988 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:01.885896 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 29 23:13:01.886419 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:01.885981 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" podUID="2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 29 23:13:02.256060 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:02.256029 5057 generic.go:356] "Generic (PLEG): container finished" podID="2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" containerID="1c2aa93d278358cb53967ac1f42e3068758e189c36e5c58fb78d1c8d8c5a6006" exitCode=0 Oct 29 23:13:02.256275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:02.256078 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" event={"ID":"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276","Type":"ContainerDied","Data":"1c2aa93d278358cb53967ac1f42e3068758e189c36e5c58fb78d1c8d8c5a6006"} Oct 29 23:13:02.256616 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:02.256598 5057 scope.go:122] "RemoveContainer" containerID="1c2aa93d278358cb53967ac1f42e3068758e189c36e5c58fb78d1c8d8c5a6006" Oct 29 23:13:02.258356 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:02.258326 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:02.258460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:02.258394 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:02.258460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:02.258431 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:03.304210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:03.298048 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" event={"ID":"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276","Type":"ContainerStarted","Data":"c8dba7952c9d4f9e54635fb77eb4acbfd8f26c7b26c12f5da5b2a66661c14a94"} Oct 29 23:13:03.915514 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:13:03.915473 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:13:04.171607 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:04.171522 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:13:04.171798 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:04.171774 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 29 23:13:04.171869 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:04.171833 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" podUID="2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 29 23:13:05.171762 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:05.171715 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 29 23:13:05.172272 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:05.171786 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" podUID="2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 29 23:13:06.171751 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:06.171717 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 29 23:13:06.171957 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:06.171770 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" podUID="2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 29 23:13:07.171683 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:07.171646 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 29 23:13:07.171910 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:07.171695 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" podUID="2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 29 23:13:08.172742 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:08.172700 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 29 23:13:08.173172 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:08.172770 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" podUID="2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 29 23:13:09.176538 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.176503 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:13:09.177026 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.176928 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:13:09.177575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.177552 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:13:09.229553 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.229516 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f"] Oct 29 23:13:09.230605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.230578 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="34c31c14-402a-4cff-a9e1-0a6940ffc05a" containerName="istio-proxy" Oct 29 23:13:09.236090 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.236063 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" Oct 29 23:13:09.254139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.254109 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-2126fb-serving-cert\"" Oct 29 23:13:09.264929 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.264899 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f"] Oct 29 23:13:09.265852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.265822 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21544662-1c4e-4386-a123-8407a2f390d4-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f\" (UID: \"21544662-1c4e-4386-a123-8407a2f390d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" Oct 29 23:13:09.266032 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.266009 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21544662-1c4e-4386-a123-8407a2f390d4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f\" (UID: \"21544662-1c4e-4386-a123-8407a2f390d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" Oct 29 23:13:09.266210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.266169 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21544662-1c4e-4386-a123-8407a2f390d4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f\" (UID: \"21544662-1c4e-4386-a123-8407a2f390d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" Oct 29 23:13:09.266761 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.266741 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d295\" (UniqueName: \"kubernetes.io/projected/21544662-1c4e-4386-a123-8407a2f390d4-kube-api-access-4d295\") pod \"maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f\" (UID: \"21544662-1c4e-4386-a123-8407a2f390d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" Oct 29 23:13:09.371259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.371181 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d295\" (UniqueName: \"kubernetes.io/projected/21544662-1c4e-4386-a123-8407a2f390d4-kube-api-access-4d295\") pod \"maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f\" (UID: \"21544662-1c4e-4386-a123-8407a2f390d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" Oct 29 23:13:09.371744 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.371712 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21544662-1c4e-4386-a123-8407a2f390d4-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f\" (UID: \"21544662-1c4e-4386-a123-8407a2f390d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" Oct 29 23:13:09.371898 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.371878 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21544662-1c4e-4386-a123-8407a2f390d4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f\" (UID: \"21544662-1c4e-4386-a123-8407a2f390d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" Oct 29 23:13:09.372055 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.372038 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21544662-1c4e-4386-a123-8407a2f390d4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f\" (UID: \"21544662-1c4e-4386-a123-8407a2f390d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" Oct 29 23:13:09.372961 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.372757 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21544662-1c4e-4386-a123-8407a2f390d4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f\" (UID: \"21544662-1c4e-4386-a123-8407a2f390d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" Oct 29 23:13:09.374666 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.374649 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21544662-1c4e-4386-a123-8407a2f390d4-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f\" (UID: \"21544662-1c4e-4386-a123-8407a2f390d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" Oct 29 23:13:09.375726 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.375694 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21544662-1c4e-4386-a123-8407a2f390d4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f\" (UID: \"21544662-1c4e-4386-a123-8407a2f390d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" Oct 29 23:13:09.408012 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.407581 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d295\" (UniqueName: \"kubernetes.io/projected/21544662-1c4e-4386-a123-8407a2f390d4-kube-api-access-4d295\") pod \"maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f\" (UID: \"21544662-1c4e-4386-a123-8407a2f390d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" Oct 29 23:13:09.564713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.564630 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" Oct 29 23:13:09.925863 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:09.925829 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f"] Oct 29 23:13:09.926008 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:13:09.925833 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21544662_1c4e_4386_a123_8407a2f390d4.slice/crio-ffd6d1b5bb602d22d9beeef59b8d15929936073093064fbfa180b27166ceaa89 WatchSource:0}: Error finding container ffd6d1b5bb602d22d9beeef59b8d15929936073093064fbfa180b27166ceaa89: Status 404 returned error can't find the container with id ffd6d1b5bb602d22d9beeef59b8d15929936073093064fbfa180b27166ceaa89 Oct 29 23:13:10.649063 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:10.648966 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" event={"ID":"21544662-1c4e-4386-a123-8407a2f390d4","Type":"ContainerStarted","Data":"298d3b3f00c8b469bff6f896bf4a92880c5d9837675c2238f2775f78b4cf2938"} Oct 29 23:13:10.649063 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:10.649030 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" event={"ID":"21544662-1c4e-4386-a123-8407a2f390d4","Type":"ContainerStarted","Data":"ffd6d1b5bb602d22d9beeef59b8d15929936073093064fbfa180b27166ceaa89"} Oct 29 23:13:10.649063 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:10.649057 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" Oct 29 23:13:10.695133 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:10.695084 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" podStartSLOduration=1.347604859 podStartE2EDuration="1.695068727s" totalImagesPullingTime="347.463868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:10.69434872 +0000 UTC m=+1837.345847250" watchObservedRunningTime="2026-10-29 23:13:10.695068727 +0000 UTC m=+1837.346567257" Oct 29 23:13:12.050316 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.050282 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h"] Oct 29 23:13:12.056649 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.056624 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.074747 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.074542 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-4ad17e-openshift-default-dockercfg-pkm5r\"" Oct 29 23:13:12.095884 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.095844 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h"] Oct 29 23:13:12.096814 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.096794 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-podinfo\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.096895 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.096875 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-workload-socket\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.096951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.096931 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-token\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.097032 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.097016 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-envoy\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.097140 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.097122 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-credential-socket\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.097207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.097164 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvzb5\" (UniqueName: \"kubernetes.io/projected/0d81f061-78fe-4a4a-aba1-11cc32a218c6-kube-api-access-dvzb5\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.097266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.097237 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istiod-ca-cert\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.097414 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.097399 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-workload-certs\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.097522 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.097508 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-data\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.204754 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.204703 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-workload-certs\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.204959 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.204827 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-data\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.204959 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.204878 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-podinfo\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.204959 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.204929 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-workload-socket\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.205145 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.204981 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-token\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.205145 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.205077 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-envoy\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.205322 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.205145 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-credential-socket\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.205322 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.205181 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvzb5\" (UniqueName: \"kubernetes.io/projected/0d81f061-78fe-4a4a-aba1-11cc32a218c6-kube-api-access-dvzb5\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.205322 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.205236 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istiod-ca-cert\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.225229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.205986 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-credential-socket\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.225229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.212417 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-data\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.225229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.212866 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-workload-certs\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.225229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.218318 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istiod-ca-cert\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.227165 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.227127 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-workload-socket\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.244434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.244397 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-envoy\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.295208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.285426 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvzb5\" (UniqueName: \"kubernetes.io/projected/0d81f061-78fe-4a4a-aba1-11cc32a218c6-kube-api-access-dvzb5\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.308289 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.303836 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-podinfo\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.347211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.345429 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-token\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.381373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.381029 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:12.755026 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:13:12.754989 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d81f061_78fe_4a4a_aba1_11cc32a218c6.slice/crio-149d72144527ae5d4365e3ec77d33675020c8228c3d8342e2acb840ccbc11052 WatchSource:0}: Error finding container 149d72144527ae5d4365e3ec77d33675020c8228c3d8342e2acb840ccbc11052: Status 404 returned error can't find the container with id 149d72144527ae5d4365e3ec77d33675020c8228c3d8342e2acb840ccbc11052 Oct 29 23:13:12.757681 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.757645 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:12.757808 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.757720 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:12.757808 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.757762 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:12.767813 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:12.767786 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h"] Oct 29 23:13:13.717756 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:13.717716 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" event={"ID":"0d81f061-78fe-4a4a-aba1-11cc32a218c6","Type":"ContainerStarted","Data":"41ae0ebf6b52de4ab6e7ddd0f3107404a7b0a677997eec68765b437e49f80efe"} Oct 29 23:13:13.718228 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:13.717774 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" event={"ID":"0d81f061-78fe-4a4a-aba1-11cc32a218c6","Type":"ContainerStarted","Data":"149d72144527ae5d4365e3ec77d33675020c8228c3d8342e2acb840ccbc11052"} Oct 29 23:13:13.783924 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:13.783857 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" podStartSLOduration=1.783839052 podStartE2EDuration="1.783839052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:13.779911322 +0000 UTC m=+1840.431409855" watchObservedRunningTime="2026-10-29 23:13:13.783839052 +0000 UTC m=+1840.435337587" Oct 29 23:13:14.382477 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:14.382444 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:14.388866 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:14.388835 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:14.724601 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:14.724569 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:14.725833 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:14.725806 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:13:15.870949 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:15.870915 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg"] Oct 29 23:13:15.877673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:15.877647 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:15.891027 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:15.890996 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg"] Oct 29 23:13:15.907121 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:15.907083 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c83b50-openshift-default-dockercfg-r28p9\"" Oct 29 23:13:15.927460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:15.927424 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-podinfo\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:15.927781 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:15.927754 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-data\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:15.927930 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:15.927864 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-workload-socket\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:15.928003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:15.927951 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-token\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:15.928063 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:15.928027 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st2d6\" (UniqueName: \"kubernetes.io/projected/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-kube-api-access-st2d6\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:15.928063 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:15.928055 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-workload-certs\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:15.928234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:15.928108 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-envoy\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:15.928234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:15.928136 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istiod-ca-cert\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:15.928405 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:15.928262 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-credential-socket\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:16.035399 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:16.035305 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-token\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:16.044220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:16.036094 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st2d6\" (UniqueName: \"kubernetes.io/projected/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-kube-api-access-st2d6\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:16.044875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:16.044847 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-workload-certs\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:16.065983 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:16.065953 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-envoy\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:16.081218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:16.066515 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istiod-ca-cert\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:16.081218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:16.066673 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-credential-socket\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:16.081218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:16.066817 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-podinfo\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:16.081218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:16.066859 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-data\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:16.081218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:16.066988 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-workload-socket\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:16.081218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:16.067449 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-workload-socket\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:16.081218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:16.068077 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istiod-ca-cert\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:16.083425 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:16.082991 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-data\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:16.084666 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:16.083733 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-credential-socket\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:16.084666 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:16.065865 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-workload-certs\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:16.170261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:16.166417 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-podinfo\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:16.209815 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:16.185884 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-envoy\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:16.233207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:16.210905 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-token\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:16.239059 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:16.238987 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st2d6\" (UniqueName: \"kubernetes.io/projected/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-kube-api-access-st2d6\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:16.506390 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:16.506352 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:16.906262 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:16.902423 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" Oct 29 23:13:17.062764 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:13:17.062728 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfdb32d_7346_42ac_a5e6_3081f0895a8e.slice/crio-6bb1b6c5233c10388ddaba2fc2018981a8ce02c727bcbaa2ebc66d33bf03e543 WatchSource:0}: Error finding container 6bb1b6c5233c10388ddaba2fc2018981a8ce02c727bcbaa2ebc66d33bf03e543: Status 404 returned error can't find the container with id 6bb1b6c5233c10388ddaba2fc2018981a8ce02c727bcbaa2ebc66d33bf03e543 Oct 29 23:13:17.064089 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:17.064062 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg"] Oct 29 23:13:17.065806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:17.065770 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:17.065874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:17.065840 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:17.065874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:17.065868 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:17.957505 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:17.957457 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" event={"ID":"7dfdb32d-7346-42ac-a5e6-3081f0895a8e","Type":"ContainerStarted","Data":"d2768bcf76f84f06ec88d3a3680e42b3d04598024460cfffcfc7be28f3f8a63b"} Oct 29 23:13:17.957980 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:17.957527 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" event={"ID":"7dfdb32d-7346-42ac-a5e6-3081f0895a8e","Type":"ContainerStarted","Data":"6bb1b6c5233c10388ddaba2fc2018981a8ce02c727bcbaa2ebc66d33bf03e543"} Oct 29 23:13:18.664211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:18.649491 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:18.793765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:18.793728 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:18.982353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:18.960953 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" podStartSLOduration=3.9609300579999998 podStartE2EDuration="3.960930058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:18.035791338 +0000 UTC m=+1844.687289867" watchObservedRunningTime="2026-10-29 23:13:18.960930058 +0000 UTC m=+1845.612428588" Oct 29 23:13:19.040676 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:19.039960 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:19.041977 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:19.041673 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:13:20.326543 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:20.326502 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:13:24.218351 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:24.218298 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:24.271307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:24.271262 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 23:13:24.369217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:24.331290 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:25.228825 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:25.215656 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:25.282211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:25.278676 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:25.282211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:25.279112 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:13:25.322935 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:25.322887 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:13:25.359986 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:25.359308 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" podUID="f7feb618-94bf-4988-abca-4dc4fbd699c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:25.420212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:25.396845 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:25.977211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:25.976793 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:26.229270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:26.220459 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:26.295912 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:26.292055 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 23:13:26.324338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:26.324287 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:26.429350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:26.429297 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:26.430288 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:26.430262 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:13:26.441455 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:26.441430 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:13:27.224324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:27.224275 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:27.253210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:27.228808 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:13:27.335771 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:27.335730 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:27.347850 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:27.347819 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:13:27.457211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:27.440442 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:28.274235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:28.274202 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:13:28.281165 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:28.281135 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:13:28.401217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:28.390243 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:28.507219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:28.482584 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 23:13:29.225121 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.225080 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf"] Oct 29 23:13:29.419468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.419435 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf"] Oct 29 23:13:29.419955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.419559 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.441585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.441550 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-7fb940-openshift-default-dockercfg-bhqpg\"" Oct 29 23:13:29.494373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.494279 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:13:29.533408 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.533363 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-podinfo\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.533685 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.533665 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-credential-socket\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.533812 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.533799 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istiod-ca-cert\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.533952 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.533938 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-workload-certs\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.537222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.534073 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-workload-socket\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.537558 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.537530 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-token\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.537735 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.537717 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m49g5\" (UniqueName: \"kubernetes.io/projected/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-kube-api-access-m49g5\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.538031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.538011 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-envoy\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.538222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.538205 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-data\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.688212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.654295 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-envoy\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.688562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.688530 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-data\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.688776 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.688748 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-podinfo\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.688937 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.688918 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-credential-socket\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.713150 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.713111 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istiod-ca-cert\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.713532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.713508 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-workload-certs\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.713713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.713695 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-workload-socket\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.713973 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.713956 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-token\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.714138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.714124 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m49g5\" (UniqueName: \"kubernetes.io/projected/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-kube-api-access-m49g5\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.722213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.674537 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-envoy\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.722213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.719976 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-data\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.742093 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.728957 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-credential-socket\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.742093 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.730404 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istiod-ca-cert\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.773078 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.742837 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-workload-certs\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.773078 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.743849 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-workload-socket\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:29.951215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:29.938847 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-podinfo\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:30.058875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:30.058792 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-token\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:30.091580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:30.091543 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m49g5\" (UniqueName: \"kubernetes.io/projected/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-kube-api-access-m49g5\") pod \"e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:30.238212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:30.212592 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:30.238212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:30.214286 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:30.318549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:30.318450 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:30.320009 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:30.318804 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:30.405212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:30.397942 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:31.725211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:31.717706 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:31.725211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:31.717872 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:31.725211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:31.717978 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:32.214212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:32.214164 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf"] Oct 29 23:13:32.653118 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:32.653002 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" event={"ID":"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1","Type":"ContainerStarted","Data":"ae2682b109b3b42bb5ccc3bdb9ebcf007ae33382b7780761057bcf600af2436a"} Oct 29 23:13:32.653503 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:32.653481 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" event={"ID":"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1","Type":"ContainerStarted","Data":"d1a34fed501c6c87bec47f4dc9723d92695b911a2a6adf80a933fb5d70481518"} Oct 29 23:13:32.723441 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:32.723376 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" podStartSLOduration=4.723355973 podStartE2EDuration="4.723355973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:32.717634396 +0000 UTC m=+1859.369132927" watchObservedRunningTime="2026-10-29 23:13:32.723355973 +0000 UTC m=+1859.374854506" Oct 29 23:13:33.242141 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:33.241466 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:33.247704 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:33.247633 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:33.664410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:33.664317 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:33.665525 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:33.665501 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:13:33.925746 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:13:33.925710 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:13:36.131019 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.130979 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w"] Oct 29 23:13:36.237051 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.237016 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w"] Oct 29 23:13:36.237259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.237131 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.271879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.271844 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-credential-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.272091 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.271903 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-podinfo\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.272091 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.272021 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-envoy\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.272091 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.272068 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-workload-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.272292 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.272260 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqwnz\" (UniqueName: \"kubernetes.io/projected/11ebf457-f868-43b1-890e-5ccfb79934b4-kube-api-access-pqwnz\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.272292 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.272477 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ebf457-f868-43b1-890e-5ccfb79934b4-istiod-ca-cert\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.272292 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.272533 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-workload-certs\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.272876 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.272825 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-data\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.272934 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.272921 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-token\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.381793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.380672 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-credential-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.381793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.380733 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-podinfo\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.381793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.380776 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-envoy\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.381793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.380806 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-workload-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.381793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.380879 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqwnz\" (UniqueName: \"kubernetes.io/projected/11ebf457-f868-43b1-890e-5ccfb79934b4-kube-api-access-pqwnz\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.381793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.381071 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ebf457-f868-43b1-890e-5ccfb79934b4-istiod-ca-cert\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.381793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.381124 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-workload-certs\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.381793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.381321 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-data\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.381793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.381371 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-token\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.382492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.382038 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-credential-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.386231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.385138 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ebf457-f868-43b1-890e-5ccfb79934b4-istiod-ca-cert\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.386938 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.386546 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-workload-certs\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.386938 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.386507 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-data\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.386938 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.386901 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-workload-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.391678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.391651 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-podinfo\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.393038 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.393009 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-envoy\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.411875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.411845 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqwnz\" (UniqueName: \"kubernetes.io/projected/11ebf457-f868-43b1-890e-5ccfb79934b4-kube-api-access-pqwnz\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.412059 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.412037 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-token\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:36.636105 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:36.636023 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:37.217591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:37.217559 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w"] Oct 29 23:13:37.219488 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:13:37.219458 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ebf457_f868_43b1_890e_5ccfb79934b4.slice/crio-5a432c2f8e30566e3b49c7d8d8cc6ee9e8d97cd02d3e2c246e79e088e48c184d WatchSource:0}: Error finding container 5a432c2f8e30566e3b49c7d8d8cc6ee9e8d97cd02d3e2c246e79e088e48c184d: Status 404 returned error can't find the container with id 5a432c2f8e30566e3b49c7d8d8cc6ee9e8d97cd02d3e2c246e79e088e48c184d Oct 29 23:13:37.223351 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:37.223314 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:37.223463 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:37.223385 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:37.223463 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:37.223413 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:37.848532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:37.848484 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" event={"ID":"11ebf457-f868-43b1-890e-5ccfb79934b4","Type":"ContainerStarted","Data":"8e32623555ab55dff298e7421c1d11a529c7ce5f4039553cf8e312cd8ef23798"} Oct 29 23:13:37.849174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:37.848569 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" event={"ID":"11ebf457-f868-43b1-890e-5ccfb79934b4","Type":"ContainerStarted","Data":"5a432c2f8e30566e3b49c7d8d8cc6ee9e8d97cd02d3e2c246e79e088e48c184d"} Oct 29 23:13:37.914555 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:37.914240 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" podStartSLOduration=1.914220274 podStartE2EDuration="1.914220274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:37.906985419 +0000 UTC m=+1864.558483987" watchObservedRunningTime="2026-10-29 23:13:37.914220274 +0000 UTC m=+1864.565718805" Oct 29 23:13:38.639542 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:38.639238 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:38.653397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:38.639369 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 29 23:13:38.653397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:38.651469 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" podUID="11ebf457-f868-43b1-890e-5ccfb79934b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 29 23:13:38.667168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:38.667040 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": read tcp 10.132.0.2:53554->10.132.0.200:15021: read: connection reset by peer" start-of-body= Oct 29 23:13:38.667671 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:38.667638 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" podUID="fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": read tcp 10.132.0.2:53554->10.132.0.200:15021: read: connection reset by peer" Oct 29 23:13:38.867873 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:38.867838 5057 generic.go:356] "Generic (PLEG): container finished" podID="fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" containerID="8329e3fc2f1ef0936c81518e85af99bc6108bb21fca5b0a93f0457bca99f0dda" exitCode=0 Oct 29 23:13:38.869334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:38.869298 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" event={"ID":"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31","Type":"ContainerDied","Data":"8329e3fc2f1ef0936c81518e85af99bc6108bb21fca5b0a93f0457bca99f0dda"} Oct 29 23:13:38.869897 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:38.869877 5057 scope.go:122] "RemoveContainer" containerID="8329e3fc2f1ef0936c81518e85af99bc6108bb21fca5b0a93f0457bca99f0dda" Oct 29 23:13:38.873678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:38.873645 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:38.873787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:38.873733 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:38.873787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:38.873777 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:39.641289 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:39.640610 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 29 23:13:39.641289 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:39.640675 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" podUID="11ebf457-f868-43b1-890e-5ccfb79934b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 29 23:13:39.879021 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:39.878981 5057 generic.go:356] "Generic (PLEG): container finished" podID="9c0197f2-eb04-41dc-8a00-a3f97ee557d3" containerID="01e4a75d65e031938da18a02fab0cc284eea8825272860d642633b41a7516a67" exitCode=0 Oct 29 23:13:39.879267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:39.879051 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" event={"ID":"9c0197f2-eb04-41dc-8a00-a3f97ee557d3","Type":"ContainerDied","Data":"01e4a75d65e031938da18a02fab0cc284eea8825272860d642633b41a7516a67"} Oct 29 23:13:39.879638 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:39.879613 5057 scope.go:122] "RemoveContainer" containerID="01e4a75d65e031938da18a02fab0cc284eea8825272860d642633b41a7516a67" Oct 29 23:13:39.881518 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:39.881497 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" event={"ID":"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31","Type":"ContainerStarted","Data":"ec9191720004d77e324ed60f793e094da40056f13c29439c5012d9acd9e1ece2"} Oct 29 23:13:39.881621 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:39.881567 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:39.881679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:39.881660 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:39.881728 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:39.881705 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:39.883806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:39.883775 5057 generic.go:356] "Generic (PLEG): container finished" podID="b783e0d4-8272-4c3e-b046-0d6a70160fea" containerID="a9df19ba5de2e880e94d51add5dbaffa926cb5932164ed84a8b74f4607ad03d0" exitCode=0 Oct 29 23:13:39.883897 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:39.883841 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" event={"ID":"b783e0d4-8272-4c3e-b046-0d6a70160fea","Type":"ContainerDied","Data":"a9df19ba5de2e880e94d51add5dbaffa926cb5932164ed84a8b74f4607ad03d0"} Oct 29 23:13:39.884365 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:39.884351 5057 scope.go:122] "RemoveContainer" containerID="a9df19ba5de2e880e94d51add5dbaffa926cb5932164ed84a8b74f4607ad03d0" Oct 29 23:13:39.886030 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:39.885998 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:39.886116 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:39.886080 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:39.886166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:39.886123 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:40.204160 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:40.199568 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:13:40.204160 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:40.203620 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 23:13:40.204160 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:40.203833 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" podUID="fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 23:13:40.640797 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:40.640292 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 29 23:13:40.640797 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:40.640365 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" podUID="11ebf457-f868-43b1-890e-5ccfb79934b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 29 23:13:41.007213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:41.001122 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" event={"ID":"b783e0d4-8272-4c3e-b046-0d6a70160fea","Type":"ContainerStarted","Data":"558dd3cbd60243bb4d7f90e1fd45e441fba0cb26c962d104eaab43a80c6b87be"} Oct 29 23:13:41.045212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:41.041671 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" event={"ID":"9c0197f2-eb04-41dc-8a00-a3f97ee557d3","Type":"ContainerStarted","Data":"684c126d3c918ca8b158bd1e47d411d60f577473784a2e82fee629d9e9669113"} Oct 29 23:13:41.199940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:41.199907 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 23:13:41.200205 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:41.199962 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" podUID="fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 23:13:41.541213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:41.534960 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 29 23:13:41.541213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:41.535039 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" podUID="b783e0d4-8272-4c3e-b046-0d6a70160fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 29 23:13:41.541213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:41.531349 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:13:41.640211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:41.636608 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 29 23:13:41.640211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:41.636683 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" podUID="11ebf457-f868-43b1-890e-5ccfb79934b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 29 23:13:41.981682 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:41.981650 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:13:42.001148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:42.001101 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 29 23:13:42.006345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:42.006284 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" podUID="9c0197f2-eb04-41dc-8a00-a3f97ee557d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 29 23:13:42.205114 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:42.204652 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 23:13:42.205114 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:42.204714 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" podUID="fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 23:13:42.746698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:42.746656 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 29 23:13:42.747164 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:42.747124 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" podUID="b783e0d4-8272-4c3e-b046-0d6a70160fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 29 23:13:42.966934 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:42.966896 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 29 23:13:42.967254 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:42.967218 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" podUID="11ebf457-f868-43b1-890e-5ccfb79934b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 29 23:13:42.980420 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:42.967362 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj"] Oct 29 23:13:43.201212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.194530 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 29 23:13:43.201212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.194626 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" podUID="9c0197f2-eb04-41dc-8a00-a3f97ee557d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 29 23:13:43.392213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.327262 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:43.392213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.374492 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 23:13:43.392213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.374589 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" podUID="fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 23:13:43.392213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.374712 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:43.406213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.403320 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj"] Oct 29 23:13:43.406213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.403393 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:13:43.406213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.403531 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:43.579705 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.579645 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 29 23:13:43.579980 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.579950 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" podUID="b783e0d4-8272-4c3e-b046-0d6a70160fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 29 23:13:43.721040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.720154 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:43.721040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.720292 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r98j\" (UniqueName: \"kubernetes.io/projected/36e06ddc-5f05-453b-831a-1d533490f290-kube-api-access-6r98j\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:43.721040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.720469 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36e06ddc-5f05-453b-831a-1d533490f290-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:43.721040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.720499 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36e06ddc-5f05-453b-831a-1d533490f290-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:43.721040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.720549 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:43.721040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.720651 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:43.721040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.720787 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:43.721040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.720879 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:43.721040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.720912 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36e06ddc-5f05-453b-831a-1d533490f290-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:43.771583 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.771486 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 29 23:13:43.804385 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.804335 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" podUID="11ebf457-f868-43b1-890e-5ccfb79934b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 29 23:13:43.977248 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.973087 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36e06ddc-5f05-453b-831a-1d533490f290-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:43.977248 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:43.973151 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36e06ddc-5f05-453b-831a-1d533490f290-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:44.067903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.052092 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:44.067903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.052278 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:44.067903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.052441 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:44.067903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.052539 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:44.067903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.052574 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36e06ddc-5f05-453b-831a-1d533490f290-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:44.067903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.052804 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:44.067903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.052885 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r98j\" (UniqueName: \"kubernetes.io/projected/36e06ddc-5f05-453b-831a-1d533490f290-kube-api-access-6r98j\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:44.067903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.053951 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:44.067903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.054516 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:44.067903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.055023 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:44.082989 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.082960 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36e06ddc-5f05-453b-831a-1d533490f290-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:44.135589 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.104752 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:44.137218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.137172 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 29 23:13:44.137436 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.137404 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" podUID="9c0197f2-eb04-41dc-8a00-a3f97ee557d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 29 23:13:44.245275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.236060 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r98j\" (UniqueName: \"kubernetes.io/projected/36e06ddc-5f05-453b-831a-1d533490f290-kube-api-access-6r98j\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:44.381432 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.375052 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36e06ddc-5f05-453b-831a-1d533490f290-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:44.381432 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.375339 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:44.388332 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.388281 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 23:13:44.407217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.400556 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36e06ddc-5f05-453b-831a-1d533490f290-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:44.437212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.416951 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:13:44.437212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.417051 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:13:44.452218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.446725 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:44.551393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.551355 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 29 23:13:44.606891 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.606849 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" podUID="b783e0d4-8272-4c3e-b046-0d6a70160fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 29 23:13:44.607144 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.587512 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:44.615291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.559938 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 23:13:44.616035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.616002 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" podUID="fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 23:13:44.630501 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.630467 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:44.708233 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.708200 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 29 23:13:44.708503 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:44.708477 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" podUID="11ebf457-f868-43b1-890e-5ccfb79934b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 29 23:13:45.073541 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:45.073504 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 29 23:13:45.073860 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:45.073815 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" podUID="9c0197f2-eb04-41dc-8a00-a3f97ee557d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 29 23:13:45.255923 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:45.254652 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 23:13:45.255923 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:45.254713 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" podUID="fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 23:13:45.325261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:45.318329 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:45.325261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:45.321045 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:45.349151 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:45.340274 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:45.417378 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:45.412288 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 23:13:45.483222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:45.451287 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:45.483222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:45.451869 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:13:45.534321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:45.534134 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 29 23:13:45.534321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:45.534206 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" podUID="b783e0d4-8272-4c3e-b046-0d6a70160fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 29 23:13:45.774660 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:45.774622 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 29 23:13:45.774939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:45.774910 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" podUID="11ebf457-f868-43b1-890e-5ccfb79934b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 29 23:13:46.078790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:46.078686 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded" Oct 29 23:13:46.422476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:46.422382 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 29 23:13:46.459211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:46.434430 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" podUID="9c0197f2-eb04-41dc-8a00-a3f97ee557d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 29 23:13:46.463244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:46.462351 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 23:13:46.478207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:46.436579 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 23:13:46.595666 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:46.595619 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" podUID="fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 23:13:46.609685 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:46.430567 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:13:46.724212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:46.534869 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:46.749210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:46.724928 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:13:46.817218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:46.430526 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 23:13:46.945443 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:46.869358 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-5m4cw" podUID="5f7503b3-a2b7-4795-b78a-9f18d00b5f2f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:47.022317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:46.998076 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:13:47.022317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:46.999989 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 29 23:13:47.022317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:47.000031 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" podUID="9c0197f2-eb04-41dc-8a00-a3f97ee557d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 29 23:13:47.096347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:47.080728 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 29 23:13:47.107315 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:47.107270 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" podUID="b783e0d4-8272-4c3e-b046-0d6a70160fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 29 23:13:47.226378 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:47.226332 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:47.228374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:47.228355 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:47.245304 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:47.245279 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:13:47.313263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:47.306258 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 23:13:47.313263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:47.306317 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" podUID="fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 23:13:47.523492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:47.490958 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 23:13:47.529689 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:47.529651 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 29 23:13:47.529912 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:47.529885 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" podUID="b783e0d4-8272-4c3e-b046-0d6a70160fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 29 23:13:47.745228 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:47.722893 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:48.026808 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:47.727152 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 23:13:48.027453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:48.027399 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 23:13:48.039646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:47.784421 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj"] Oct 29 23:13:48.040011 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:48.039969 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:13:48.163791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:48.163757 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 29 23:13:48.164126 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:48.164074 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" podUID="9c0197f2-eb04-41dc-8a00-a3f97ee557d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 29 23:13:48.252396 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:48.252361 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 23:13:48.252665 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:48.252638 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" podUID="fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 23:13:48.253177 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:48.253140 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:48.253416 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:48.253393 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:48.253560 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:48.253540 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:48.577844 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:48.552424 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 29 23:13:48.577844 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:48.552496 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" podUID="b783e0d4-8272-4c3e-b046-0d6a70160fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 29 23:13:48.920584 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:48.920551 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:13:48.975252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:48.947647 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:13:49.000212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:48.979063 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" event={"ID":"36e06ddc-5f05-453b-831a-1d533490f290","Type":"ContainerStarted","Data":"bda2f03f66e05e8fe0ea7344c3ef62e663dc8d6b478bfae360691d39e301f8d8"} Oct 29 23:13:49.024572 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:49.024542 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:13:49.168971 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:49.080584 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 29 23:13:49.168971 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:49.111297 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" podUID="9c0197f2-eb04-41dc-8a00-a3f97ee557d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 29 23:13:49.459832 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:49.454999 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:13:49.574322 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:49.488288 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:13:49.687957 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:49.687920 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:13:49.857275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:49.852363 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 29 23:13:49.857275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:49.852453 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" podUID="b783e0d4-8272-4c3e-b046-0d6a70160fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 29 23:13:50.318207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:50.316579 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 29 23:13:50.318207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:50.316661 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" podUID="9c0197f2-eb04-41dc-8a00-a3f97ee557d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 29 23:13:50.508780 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:50.508721 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" podStartSLOduration=8.508699878 podStartE2EDuration="8.508699878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:50.457672167 +0000 UTC m=+1877.109170698" watchObservedRunningTime="2026-10-29 23:13:50.508699878 +0000 UTC m=+1877.160198410" Oct 29 23:13:50.550639 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:50.550603 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 29 23:13:50.567210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:50.552342 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" podUID="b783e0d4-8272-4c3e-b046-0d6a70160fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 29 23:13:50.652392 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:50.652302 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:51.000007 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:50.999962 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:13:51.007750 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:51.007716 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:13:51.656216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:51.625908 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" event={"ID":"36e06ddc-5f05-453b-831a-1d533490f290","Type":"ContainerStarted","Data":"2a89a1f048fe7456ac6d78eaac09a764c50aa613285f1c4c7985a77a368fb890"} Oct 29 23:13:51.656745 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:51.656716 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:13:51.702990 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:51.702954 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 29 23:13:51.703301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:51.703268 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" podUID="36e06ddc-5f05-453b-831a-1d533490f290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 29 23:13:51.776730 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:51.776690 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:13:52.652248 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:52.651380 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 29 23:13:52.652248 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:52.651443 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" podUID="36e06ddc-5f05-453b-831a-1d533490f290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 29 23:13:53.648600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:53.648565 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 29 23:13:53.648900 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:53.648871 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" podUID="36e06ddc-5f05-453b-831a-1d533490f290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 29 23:13:54.645669 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:54.645629 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 29 23:13:54.646342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:54.646305 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" podUID="36e06ddc-5f05-453b-831a-1d533490f290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 29 23:13:55.651914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:55.651398 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 29 23:13:55.651914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:55.651476 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" podUID="36e06ddc-5f05-453b-831a-1d533490f290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 29 23:13:56.207330 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.207288 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd"] Oct 29 23:13:56.231015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.224998 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd"] Oct 29 23:13:56.231015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.225139 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.262870 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.261779 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-8a8b61-openshift-default-dockercfg-lrtvw\"" Oct 29 23:13:56.285230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.285202 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvr8c\" (UniqueName: \"kubernetes.io/projected/408b4a17-ac05-47f5-bbcb-ed398229570c-kube-api-access-wvr8c\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.285489 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.285475 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-token\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.285601 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.285591 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-workload-certs\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.285809 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.285797 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-envoy\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.285912 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.285902 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-credential-socket\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.286011 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.286002 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-podinfo\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.286107 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.286098 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-data\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.286526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.286510 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-workload-socket\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.286659 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.286649 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408b4a17-ac05-47f5-bbcb-ed398229570c-istiod-ca-cert\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.436642 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.432800 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-envoy\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.436642 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.432864 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-credential-socket\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.436642 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.432919 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-podinfo\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.436642 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.432979 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-data\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.436642 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.433017 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-workload-socket\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.436642 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.433059 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408b4a17-ac05-47f5-bbcb-ed398229570c-istiod-ca-cert\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.436642 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.433241 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvr8c\" (UniqueName: \"kubernetes.io/projected/408b4a17-ac05-47f5-bbcb-ed398229570c-kube-api-access-wvr8c\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.436642 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.433292 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-token\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.436642 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.433323 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-workload-certs\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.436642 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.433916 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-workload-certs\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.436642 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.435675 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-workload-socket\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.436642 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.436251 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-credential-socket\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.437692 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.437376 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408b4a17-ac05-47f5-bbcb-ed398229570c-istiod-ca-cert\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.439218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.437781 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-data\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.479501 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.472029 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-podinfo\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.479501 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.472338 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-envoy\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.532212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.520759 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-token\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.532212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.521694 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvr8c\" (UniqueName: \"kubernetes.io/projected/408b4a17-ac05-47f5-bbcb-ed398229570c-kube-api-access-wvr8c\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.589715 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.589679 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:56.648638 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.648600 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 29 23:13:56.648923 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:56.648889 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" podUID="36e06ddc-5f05-453b-831a-1d533490f290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 29 23:13:57.187304 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:57.187255 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd"] Oct 29 23:13:57.188223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:57.188148 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:57.188338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:57.188234 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:57.188338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:57.188267 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:57.653259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:57.653140 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 29 23:13:57.653544 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:57.653504 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" podUID="36e06ddc-5f05-453b-831a-1d533490f290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 29 23:13:57.685394 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:57.685357 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" event={"ID":"408b4a17-ac05-47f5-bbcb-ed398229570c","Type":"ContainerStarted","Data":"0abed984960160f246c3f1ddf058d37cfd85832fe262ba396e07da3a699545ce"} Oct 29 23:13:57.685575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:57.685410 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" event={"ID":"408b4a17-ac05-47f5-bbcb-ed398229570c","Type":"ContainerStarted","Data":"33197e17df780cd9fb3713a9d2491cc5816abf25c2df0b7947ea589ad529ef5a"} Oct 29 23:13:57.742417 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:57.742346 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" podStartSLOduration=1.7423268410000001 podStartE2EDuration="1.742326841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:57.740859969 +0000 UTC m=+1884.392358550" watchObservedRunningTime="2026-10-29 23:13:57.742326841 +0000 UTC m=+1884.393825374" Oct 29 23:13:57.944211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:57.928763 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n"] Oct 29 23:13:58.000111 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.000078 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.069587 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.069445 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n"] Oct 29 23:13:58.089123 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.089065 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.089397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.089370 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpbsz\" (UniqueName: \"kubernetes.io/projected/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-kube-api-access-xpbsz\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.089641 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.089618 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.089810 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.089791 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.089954 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.089935 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.090081 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.090063 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.090279 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.090262 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.090519 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.090501 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.090705 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.090684 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.192546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.192501 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.192546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.192555 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.193122 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.192575 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.193122 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.192599 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.193122 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.192647 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.193122 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.192761 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.193122 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.192812 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.193122 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.193019 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.193122 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.193053 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpbsz\" (UniqueName: \"kubernetes.io/projected/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-kube-api-access-xpbsz\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.193122 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.193058 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.193122 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.193073 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.193692 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.193164 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.193692 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.193502 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.193692 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.193568 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.195360 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.195299 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.195472 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.195455 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.218055 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.218019 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.218180 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.218147 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpbsz\" (UniqueName: \"kubernetes.io/projected/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-kube-api-access-xpbsz\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.391249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.391166 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:13:58.593231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.590815 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:58.599107 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.599084 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:58.647251 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.647173 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 29 23:13:58.647438 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.647256 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" podUID="36e06ddc-5f05-453b-831a-1d533490f290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 29 23:13:58.695717 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.695684 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:58.696548 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.696528 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:13:58.837412 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.837316 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n"] Oct 29 23:13:58.841880 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:13:58.841851 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb171dd_4b28_4381_b1a5_7e5a25f2aefd.slice/crio-1e6c57a923cdfc34e5472bebe5a86d19ae40dfee51596f99a3666bdf9e49252e WatchSource:0}: Error finding container 1e6c57a923cdfc34e5472bebe5a86d19ae40dfee51596f99a3666bdf9e49252e: Status 404 returned error can't find the container with id 1e6c57a923cdfc34e5472bebe5a86d19ae40dfee51596f99a3666bdf9e49252e Oct 29 23:13:58.884055 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.884008 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:58.884284 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.884102 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:58.884284 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:58.884152 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:13:59.645559 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:59.645521 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:13:59.704101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:59.704059 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" event={"ID":"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd","Type":"ContainerStarted","Data":"bb48e2e1ff4fb066d0f52cbbc4ac90cd2fd3d6ee5ccbe70cfb19cb3eb3d3eb90"} Oct 29 23:13:59.704331 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:59.704123 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" event={"ID":"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd","Type":"ContainerStarted","Data":"1e6c57a923cdfc34e5472bebe5a86d19ae40dfee51596f99a3666bdf9e49252e"} Oct 29 23:13:59.791445 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:13:59.791391 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" podStartSLOduration=2.791375297 podStartE2EDuration="2.791375297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:59.775164779 +0000 UTC m=+1886.426663315" watchObservedRunningTime="2026-10-29 23:13:59.791375297 +0000 UTC m=+1886.442873826" Oct 29 23:14:00.403763 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:00.403730 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:14:00.414226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:00.407969 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 29 23:14:00.414226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:00.408039 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" podUID="8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 29 23:14:00.916212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:00.912253 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:14:00.916212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:00.912323 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" podUID="36e06ddc-5f05-453b-831a-1d533490f290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:14:01.404651 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:01.404612 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 29 23:14:01.404869 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:01.404693 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" podUID="8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 29 23:14:01.472119 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:01.472092 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:14:01.485101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:01.485074 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:14:02.435209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:02.413371 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 29 23:14:02.435875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:02.435834 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" podUID="8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 29 23:14:03.420240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:03.408698 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 29 23:14:03.420240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:03.408776 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" podUID="8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 29 23:14:03.916702 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:14:03.916663 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:14:04.407239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:04.404397 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 29 23:14:04.407239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:04.404474 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" podUID="8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 29 23:14:05.446212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:05.434322 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 29 23:14:05.446212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:05.434395 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" podUID="8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 29 23:14:06.405209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:06.404588 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 29 23:14:06.405209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:06.404661 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" podUID="8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 29 23:14:07.403632 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:07.403590 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 29 23:14:07.404036 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:07.403666 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" podUID="8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 29 23:14:08.409034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:08.409005 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:14:08.409476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:08.409454 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:14:08.410142 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:08.410123 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:14:14.169423 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:14.169369 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf"] Oct 29 23:14:14.170605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:14.170537 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" podUID="f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1" containerName="istio-proxy" containerID="cri-o://ae2682b109b3b42bb5ccc3bdb9ebcf007ae33382b7780761057bcf600af2436a" gracePeriod=30 Oct 29 23:14:18.666482 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:18.666435 5057 patch_prober.go:29] interesting pod/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:14:18.666993 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:18.666508 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" podUID="f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:14:19.519127 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.519103 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:14:19.663915 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.663873 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-workload-socket\") pod \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " Oct 29 23:14:19.664115 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.663946 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-data\") pod \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " Oct 29 23:14:19.664115 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.663989 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-token\" (UniqueName: \"kubernetes.io/projected/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-token\") pod \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " Oct 29 23:14:19.664115 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.664038 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-envoy\") pod \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " Oct 29 23:14:19.664115 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.664073 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-workload-certs\") pod \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " Oct 29 23:14:19.664373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.664117 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-kube-api-access-m49g5\" (UniqueName: \"kubernetes.io/projected/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-kube-api-access-m49g5\") pod \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " Oct 29 23:14:19.664373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.664168 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istiod-ca-cert\") pod \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " Oct 29 23:14:19.664373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.664227 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-credential-socket\") pod \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " Oct 29 23:14:19.664373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.664284 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-podinfo\") pod \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\" (UID: \"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1\") " Oct 29 23:14:19.664590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.664432 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-workload-certs" pod "f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1" (UID: "f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:19.664879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.664853 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-data" pod "f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1" (UID: "f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:19.665180 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.665144 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:14:19.665180 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.665173 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:14:19.665611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.665588 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istiod-ca-cert" pod "f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1" (UID: "f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:14:19.666260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.666231 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-credential-socket" pod "f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1" (UID: "f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:19.666838 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.666814 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-workload-socket" pod "f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1" (UID: "f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:19.668456 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.668422 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-token" pod "f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1" (UID: "f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:14:19.669013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.668990 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-podinfo" pod "f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1" (UID: "f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:14:19.671946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.671925 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-envoy" pod "f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1" (UID: "f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:19.672141 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.672120 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-kube-api-access-m49g5" pod "f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1" (UID: "f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1"). InnerVolumeSpecName "kube-api-access-m49g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:14:19.767339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.767254 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m49g5\" (UniqueName: \"kubernetes.io/projected/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-kube-api-access-m49g5\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:14:19.767339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.767289 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:14:19.767339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.767303 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:14:19.767339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.767318 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:14:19.767339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.767332 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:14:19.767716 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.767344 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:14:19.767716 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:19.767357 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:14:20.294209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:20.293510 5057 generic.go:356] "Generic (PLEG): container finished" podID="f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1" containerID="ae2682b109b3b42bb5ccc3bdb9ebcf007ae33382b7780761057bcf600af2436a" exitCode=0 Oct 29 23:14:20.294209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:20.293816 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" event={"ID":"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1","Type":"ContainerDied","Data":"ae2682b109b3b42bb5ccc3bdb9ebcf007ae33382b7780761057bcf600af2436a"} Oct 29 23:14:20.294209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:20.293888 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" event={"ID":"f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1","Type":"ContainerDied","Data":"d1a34fed501c6c87bec47f4dc9723d92695b911a2a6adf80a933fb5d70481518"} Oct 29 23:14:20.294209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:20.293923 5057 scope.go:122] "RemoveContainer" containerID="ae2682b109b3b42bb5ccc3bdb9ebcf007ae33382b7780761057bcf600af2436a" Oct 29 23:14:20.294209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:20.294115 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf" Oct 29 23:14:20.338599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:20.338567 5057 scope.go:122] "RemoveContainer" containerID="ae2682b109b3b42bb5ccc3bdb9ebcf007ae33382b7780761057bcf600af2436a" Oct 29 23:14:20.338933 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:14:20.338908 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2682b109b3b42bb5ccc3bdb9ebcf007ae33382b7780761057bcf600af2436a\": container with ID starting with ae2682b109b3b42bb5ccc3bdb9ebcf007ae33382b7780761057bcf600af2436a not found: ID does not exist" containerID="ae2682b109b3b42bb5ccc3bdb9ebcf007ae33382b7780761057bcf600af2436a" Oct 29 23:14:20.339007 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:20.338945 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2682b109b3b42bb5ccc3bdb9ebcf007ae33382b7780761057bcf600af2436a"} err="failed to get container status \"ae2682b109b3b42bb5ccc3bdb9ebcf007ae33382b7780761057bcf600af2436a\": rpc error: code = NotFound desc = could not find container \"ae2682b109b3b42bb5ccc3bdb9ebcf007ae33382b7780761057bcf600af2436a\": container with ID starting with ae2682b109b3b42bb5ccc3bdb9ebcf007ae33382b7780761057bcf600af2436a not found: ID does not exist" Oct 29 23:14:20.389751 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:20.389726 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf"] Oct 29 23:14:20.403638 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:20.403614 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-7fb940-openshift-default-6f656fd8f7-gz6sf"] Oct 29 23:14:21.491099 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:21.491066 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1" path="/var/lib/kubelet/pods/f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1/volumes" Oct 29 23:14:21.765437 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:21.765343 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7"] Oct 29 23:14:21.766241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:21.766219 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1" containerName="istio-proxy" Oct 29 23:14:21.766241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:21.766241 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1" containerName="istio-proxy" Oct 29 23:14:21.766455 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:21.766441 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c47f3e-f7ae-439c-a757-ce5beb1f7aa1" containerName="istio-proxy" Oct 29 23:14:21.770288 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:21.770246 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" Oct 29 23:14:21.799439 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:21.799399 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-52f67b-serving-cert\"" Oct 29 23:14:21.805709 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:21.805677 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7"] Oct 29 23:14:21.810742 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:21.810712 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88075736-9838-476b-8cdd-36014547ca59-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7\" (UID: \"88075736-9838-476b-8cdd-36014547ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" Oct 29 23:14:21.810907 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:21.810857 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88075736-9838-476b-8cdd-36014547ca59-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7\" (UID: \"88075736-9838-476b-8cdd-36014547ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" Oct 29 23:14:21.810976 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:21.810920 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88075736-9838-476b-8cdd-36014547ca59-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7\" (UID: \"88075736-9838-476b-8cdd-36014547ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" Oct 29 23:14:21.810976 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:21.810962 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slmps\" (UniqueName: \"kubernetes.io/projected/88075736-9838-476b-8cdd-36014547ca59-kube-api-access-slmps\") pod \"maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7\" (UID: \"88075736-9838-476b-8cdd-36014547ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" Oct 29 23:14:21.912425 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:21.912391 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88075736-9838-476b-8cdd-36014547ca59-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7\" (UID: \"88075736-9838-476b-8cdd-36014547ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" Oct 29 23:14:21.912620 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:21.912517 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88075736-9838-476b-8cdd-36014547ca59-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7\" (UID: \"88075736-9838-476b-8cdd-36014547ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" Oct 29 23:14:21.912620 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:21.912598 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88075736-9838-476b-8cdd-36014547ca59-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7\" (UID: \"88075736-9838-476b-8cdd-36014547ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" Oct 29 23:14:21.912741 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:21.912653 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slmps\" (UniqueName: \"kubernetes.io/projected/88075736-9838-476b-8cdd-36014547ca59-kube-api-access-slmps\") pod \"maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7\" (UID: \"88075736-9838-476b-8cdd-36014547ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" Oct 29 23:14:21.913289 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:21.913260 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88075736-9838-476b-8cdd-36014547ca59-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7\" (UID: \"88075736-9838-476b-8cdd-36014547ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" Oct 29 23:14:21.914963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:21.914939 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88075736-9838-476b-8cdd-36014547ca59-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7\" (UID: \"88075736-9838-476b-8cdd-36014547ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" Oct 29 23:14:21.915097 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:21.915024 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88075736-9838-476b-8cdd-36014547ca59-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7\" (UID: \"88075736-9838-476b-8cdd-36014547ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" Oct 29 23:14:21.939071 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:21.939045 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slmps\" (UniqueName: \"kubernetes.io/projected/88075736-9838-476b-8cdd-36014547ca59-kube-api-access-slmps\") pod \"maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7\" (UID: \"88075736-9838-476b-8cdd-36014547ca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" Oct 29 23:14:22.086650 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:22.086493 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" Oct 29 23:14:22.501951 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:14:22.501919 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88075736_9838_476b_8cdd_36014547ca59.slice/crio-5a1cb6a986d9e19677deb9de9b7edc86732e9e89a138c6ab3728b4ef258a0d3c WatchSource:0}: Error finding container 5a1cb6a986d9e19677deb9de9b7edc86732e9e89a138c6ab3728b4ef258a0d3c: Status 404 returned error can't find the container with id 5a1cb6a986d9e19677deb9de9b7edc86732e9e89a138c6ab3728b4ef258a0d3c Oct 29 23:14:22.502443 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:22.502394 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7"] Oct 29 23:14:23.324039 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:23.323998 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" event={"ID":"88075736-9838-476b-8cdd-36014547ca59","Type":"ContainerStarted","Data":"48b808432e96653ca28663b536501484881cccb44cb4f07b7b053821852a72f4"} Oct 29 23:14:23.324303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:23.324064 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" event={"ID":"88075736-9838-476b-8cdd-36014547ca59","Type":"ContainerStarted","Data":"5a1cb6a986d9e19677deb9de9b7edc86732e9e89a138c6ab3728b4ef258a0d3c"} Oct 29 23:14:23.324303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:23.324093 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" Oct 29 23:14:23.373229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:23.373162 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" podStartSLOduration=1.981124508 podStartE2EDuration="2.373148303s" totalImagesPullingTime="392.023795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:23.369305905 +0000 UTC m=+1910.020804437" watchObservedRunningTime="2026-10-29 23:14:23.373148303 +0000 UTC m=+1910.024646834" Oct 29 23:14:29.339830 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:29.339796 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" Oct 29 23:14:29.471157 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:29.471119 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:14:33.979000 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:14:33.978953 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:14:37.081023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:37.080985 5057 scope.go:122] "RemoveContainer" containerID="45e2ae4f8f200c2565f0f66877818df0ca5b7e12c08cdac04c600d6f45f52630" Oct 29 23:14:42.271290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:42.271248 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6"] Oct 29 23:14:42.348493 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:42.348460 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6"] Oct 29 23:14:42.348912 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:42.348831 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6" Oct 29 23:14:42.393245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:42.393176 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-2591f6\"" Oct 29 23:14:42.478230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:42.448787 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-2591f6-dockercfg-zx5bg\"" Oct 29 23:14:42.535062 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:42.504199 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6225f8f9-8cbb-46cb-9563-311fab3d5e21-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6\" (UID: \"6225f8f9-8cbb-46cb-9563-311fab3d5e21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6" Oct 29 23:14:42.535062 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:42.534789 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-787cv\" (UniqueName: \"kubernetes.io/projected/6225f8f9-8cbb-46cb-9563-311fab3d5e21-kube-api-access-787cv\") pod \"payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6\" (UID: \"6225f8f9-8cbb-46cb-9563-311fab3d5e21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6" Oct 29 23:14:42.662418 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:42.638428 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-787cv\" (UniqueName: \"kubernetes.io/projected/6225f8f9-8cbb-46cb-9563-311fab3d5e21-kube-api-access-787cv\") pod \"payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6\" (UID: \"6225f8f9-8cbb-46cb-9563-311fab3d5e21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6" Oct 29 23:14:42.676225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:42.662861 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6225f8f9-8cbb-46cb-9563-311fab3d5e21-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6\" (UID: \"6225f8f9-8cbb-46cb-9563-311fab3d5e21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6" Oct 29 23:14:42.676225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:42.665330 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6225f8f9-8cbb-46cb-9563-311fab3d5e21-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6\" (UID: \"6225f8f9-8cbb-46cb-9563-311fab3d5e21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6" Oct 29 23:14:43.040368 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:43.040326 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-787cv\" (UniqueName: \"kubernetes.io/projected/6225f8f9-8cbb-46cb-9563-311fab3d5e21-kube-api-access-787cv\") pod \"payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6\" (UID: \"6225f8f9-8cbb-46cb-9563-311fab3d5e21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6" Oct 29 23:14:43.153840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:43.153613 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6" Oct 29 23:14:43.555214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:43.554240 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6"] Oct 29 23:14:44.085993 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:44.085952 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6" event={"ID":"6225f8f9-8cbb-46cb-9563-311fab3d5e21","Type":"ContainerStarted","Data":"ac83edbf62004f98d5dbaba36a936d9b5a6507c54af824996d27f14c4553cf47"} Oct 29 23:14:44.088389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:44.086793 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6" event={"ID":"6225f8f9-8cbb-46cb-9563-311fab3d5e21","Type":"ContainerStarted","Data":"884de911b0e399207d48302396c86ae31c55667b72df2da801267dc2394ee33b"} Oct 29 23:14:44.088603 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:44.088580 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6" Oct 29 23:14:44.133160 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:44.133093 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6" podStartSLOduration=2.133073152 podStartE2EDuration="2.133073152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:44.130005204 +0000 UTC m=+1930.781503735" watchObservedRunningTime="2026-10-29 23:14:44.133073152 +0000 UTC m=+1930.784571682" Oct 29 23:14:45.181990 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:45.180034 5057 generic.go:356] "Generic (PLEG): container finished" podID="7dfdb32d-7346-42ac-a5e6-3081f0895a8e" containerID="d2768bcf76f84f06ec88d3a3680e42b3d04598024460cfffcfc7be28f3f8a63b" exitCode=0 Oct 29 23:14:45.181990 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:45.180333 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" event={"ID":"7dfdb32d-7346-42ac-a5e6-3081f0895a8e","Type":"ContainerDied","Data":"d2768bcf76f84f06ec88d3a3680e42b3d04598024460cfffcfc7be28f3f8a63b"} Oct 29 23:14:45.181990 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:45.181308 5057 scope.go:122] "RemoveContainer" containerID="d2768bcf76f84f06ec88d3a3680e42b3d04598024460cfffcfc7be28f3f8a63b" Oct 29 23:14:45.184879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:45.184839 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:14:45.185137 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:45.185116 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:14:45.185319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:45.185299 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:14:45.767366 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:45.767323 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:14:45.767576 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:45.767397 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" podUID="0d81f061-78fe-4a4a-aba1-11cc32a218c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:14:46.198382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:46.198341 5057 generic.go:356] "Generic (PLEG): container finished" podID="0d81f061-78fe-4a4a-aba1-11cc32a218c6" containerID="41ae0ebf6b52de4ab6e7ddd0f3107404a7b0a677997eec68765b437e49f80efe" exitCode=0 Oct 29 23:14:46.198879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:46.198419 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" event={"ID":"0d81f061-78fe-4a4a-aba1-11cc32a218c6","Type":"ContainerDied","Data":"41ae0ebf6b52de4ab6e7ddd0f3107404a7b0a677997eec68765b437e49f80efe"} Oct 29 23:14:46.199124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:46.199094 5057 scope.go:122] "RemoveContainer" containerID="41ae0ebf6b52de4ab6e7ddd0f3107404a7b0a677997eec68765b437e49f80efe" Oct 29 23:14:46.201945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:46.201637 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:14:46.201945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:46.201750 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:14:46.201945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:46.201809 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:14:46.204367 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:46.203389 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" event={"ID":"7dfdb32d-7346-42ac-a5e6-3081f0895a8e","Type":"ContainerStarted","Data":"5753df527098f71141bbba9f5aa3786eb655ddf1672fc0c868586eea8679fc75"} Oct 29 23:14:46.506736 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:46.506704 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:14:46.510062 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:46.510009 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 29 23:14:46.510482 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:46.510453 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" podUID="7dfdb32d-7346-42ac-a5e6-3081f0895a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 29 23:14:47.283216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:47.283027 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" event={"ID":"0d81f061-78fe-4a4a-aba1-11cc32a218c6","Type":"ContainerStarted","Data":"a2557cbb2efa64eb45f1b7c00ce76c5ed038779c1a6d824ffab2d7e8cd9e9e38"} Oct 29 23:14:47.385346 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:47.383947 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:14:47.385346 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:47.384285 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 29 23:14:47.385346 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:47.384335 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" podUID="0d81f061-78fe-4a4a-aba1-11cc32a218c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 29 23:14:47.506905 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:47.506870 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 29 23:14:47.507080 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:47.506916 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" podUID="7dfdb32d-7346-42ac-a5e6-3081f0895a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 29 23:14:48.383082 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:48.383039 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 29 23:14:48.383546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:48.383103 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" podUID="0d81f061-78fe-4a4a-aba1-11cc32a218c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 29 23:14:48.506715 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:48.506682 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 29 23:14:48.506918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:48.506738 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" podUID="7dfdb32d-7346-42ac-a5e6-3081f0895a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 29 23:14:49.040305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:49.040268 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:14:49.382299 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:49.382180 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 29 23:14:49.382591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:49.382544 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" podUID="0d81f061-78fe-4a4a-aba1-11cc32a218c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 29 23:14:49.507232 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:49.506772 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 29 23:14:49.507232 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:49.506831 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" podUID="7dfdb32d-7346-42ac-a5e6-3081f0895a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 29 23:14:50.395674 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:50.395642 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 29 23:14:50.407587 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:50.405859 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" podUID="0d81f061-78fe-4a4a-aba1-11cc32a218c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 29 23:14:50.507269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:50.507236 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 29 23:14:50.507772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:50.507291 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" podUID="7dfdb32d-7346-42ac-a5e6-3081f0895a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 29 23:14:51.382286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:51.382240 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 29 23:14:51.382509 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:51.382320 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" podUID="0d81f061-78fe-4a4a-aba1-11cc32a218c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 29 23:14:51.507095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:51.507064 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 29 23:14:51.507281 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:51.507112 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" podUID="7dfdb32d-7346-42ac-a5e6-3081f0895a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 29 23:14:52.382471 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:52.382433 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 29 23:14:52.382950 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:52.382505 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" podUID="0d81f061-78fe-4a4a-aba1-11cc32a218c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 29 23:14:52.511481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:52.511448 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:14:52.512751 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:52.512725 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:14:53.387085 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:53.387055 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:14:53.387588 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:53.387546 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:14:53.388306 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:53.388281 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:14:56.206528 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:14:56.206501 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6" Oct 29 23:15:00.164410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.164371 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg"] Oct 29 23:15:00.169679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.169649 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" Oct 29 23:15:00.198825 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.198792 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw"] Oct 29 23:15:00.206428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.206395 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t"] Oct 29 23:15:00.207866 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.207299 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" Oct 29 23:15:00.212023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.211987 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg"] Oct 29 23:15:00.212244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.212097 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t" Oct 29 23:15:00.226786 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.226106 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh"] Oct 29 23:15:00.231694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.231670 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh" Oct 29 23:15:00.231936 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.231761 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw"] Oct 29 23:15:00.239235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.238513 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t"] Oct 29 23:15:00.265298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.265265 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh"] Oct 29 23:15:00.272577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.271817 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm"] Oct 29 23:15:00.285730 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.277791 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" Oct 29 23:15:00.285730 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.284260 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:15:00.294241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.293393 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68"] Oct 29 23:15:00.312295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.312114 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" Oct 29 23:15:00.329049 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.329011 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm"] Oct 29 23:15:00.349911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.349871 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483a170f-cb23-4c9c-9d95-47a7be2344ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw\" (UID: \"483a170f-cb23-4c9c-9d95-47a7be2344ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" Oct 29 23:15:00.351876 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.351841 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67mnz\" (UniqueName: \"kubernetes.io/projected/a53b0b99-b590-4aff-aef5-4793abe1f1d4-kube-api-access-67mnz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg\" (UID: \"a53b0b99-b590-4aff-aef5-4793abe1f1d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" Oct 29 23:15:00.352022 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.351957 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwbw2\" (UniqueName: \"kubernetes.io/projected/5f4756a4-cf22-42c0-89b7-c861adeed9ca-kube-api-access-wwbw2\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh\" (UID: \"5f4756a4-cf22-42c0-89b7-c861adeed9ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh" Oct 29 23:15:00.352090 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.352044 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dnxk\" (UniqueName: \"kubernetes.io/projected/483a170f-cb23-4c9c-9d95-47a7be2344ff-kube-api-access-4dnxk\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw\" (UID: \"483a170f-cb23-4c9c-9d95-47a7be2344ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" Oct 29 23:15:00.352090 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.352075 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a89a235c-5f5e-4b9f-b0b4-1bd345e2afae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t\" (UID: \"a89a235c-5f5e-4b9f-b0b4-1bd345e2afae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t" Oct 29 23:15:00.352238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.352106 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53b0b99-b590-4aff-aef5-4793abe1f1d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg\" (UID: \"a53b0b99-b590-4aff-aef5-4793abe1f1d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" Oct 29 23:15:00.352238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.352151 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f4756a4-cf22-42c0-89b7-c861adeed9ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh\" (UID: \"5f4756a4-cf22-42c0-89b7-c861adeed9ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh" Oct 29 23:15:00.352238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.352209 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvxtf\" (UniqueName: \"kubernetes.io/projected/a89a235c-5f5e-4b9f-b0b4-1bd345e2afae-kube-api-access-kvxtf\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t\" (UID: \"a89a235c-5f5e-4b9f-b0b4-1bd345e2afae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t" Oct 29 23:15:00.352692 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.352665 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68"] Oct 29 23:15:00.454024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.453975 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67mnz\" (UniqueName: \"kubernetes.io/projected/a53b0b99-b590-4aff-aef5-4793abe1f1d4-kube-api-access-67mnz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg\" (UID: \"a53b0b99-b590-4aff-aef5-4793abe1f1d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" Oct 29 23:15:00.454264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.454075 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f87mb\" (UniqueName: \"kubernetes.io/projected/028e4624-bd70-4f79-8026-a998ba9b9f58-kube-api-access-f87mb\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68\" (UID: \"028e4624-bd70-4f79-8026-a998ba9b9f58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" Oct 29 23:15:00.454264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.454104 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwbw2\" (UniqueName: \"kubernetes.io/projected/5f4756a4-cf22-42c0-89b7-c861adeed9ca-kube-api-access-wwbw2\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh\" (UID: \"5f4756a4-cf22-42c0-89b7-c861adeed9ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh" Oct 29 23:15:00.454264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.454137 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/028e4624-bd70-4f79-8026-a998ba9b9f58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68\" (UID: \"028e4624-bd70-4f79-8026-a998ba9b9f58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" Oct 29 23:15:00.455722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.454724 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dnxk\" (UniqueName: \"kubernetes.io/projected/483a170f-cb23-4c9c-9d95-47a7be2344ff-kube-api-access-4dnxk\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw\" (UID: \"483a170f-cb23-4c9c-9d95-47a7be2344ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" Oct 29 23:15:00.455722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.454766 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a89a235c-5f5e-4b9f-b0b4-1bd345e2afae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t\" (UID: \"a89a235c-5f5e-4b9f-b0b4-1bd345e2afae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t" Oct 29 23:15:00.455722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.454793 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53b0b99-b590-4aff-aef5-4793abe1f1d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg\" (UID: \"a53b0b99-b590-4aff-aef5-4793abe1f1d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" Oct 29 23:15:00.455722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.454833 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f4756a4-cf22-42c0-89b7-c861adeed9ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh\" (UID: \"5f4756a4-cf22-42c0-89b7-c861adeed9ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh" Oct 29 23:15:00.455722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.454869 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvxtf\" (UniqueName: \"kubernetes.io/projected/a89a235c-5f5e-4b9f-b0b4-1bd345e2afae-kube-api-access-kvxtf\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t\" (UID: \"a89a235c-5f5e-4b9f-b0b4-1bd345e2afae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t" Oct 29 23:15:00.455722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.454995 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35ad588a-24ca-4823-9d3d-3f281139f7e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm\" (UID: \"35ad588a-24ca-4823-9d3d-3f281139f7e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" Oct 29 23:15:00.455722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.455070 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4td9f\" (UniqueName: \"kubernetes.io/projected/35ad588a-24ca-4823-9d3d-3f281139f7e6-kube-api-access-4td9f\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm\" (UID: \"35ad588a-24ca-4823-9d3d-3f281139f7e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" Oct 29 23:15:00.455722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.455182 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483a170f-cb23-4c9c-9d95-47a7be2344ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw\" (UID: \"483a170f-cb23-4c9c-9d95-47a7be2344ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" Oct 29 23:15:00.456353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.456060 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483a170f-cb23-4c9c-9d95-47a7be2344ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw\" (UID: \"483a170f-cb23-4c9c-9d95-47a7be2344ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" Oct 29 23:15:00.456813 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.456787 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a89a235c-5f5e-4b9f-b0b4-1bd345e2afae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t\" (UID: \"a89a235c-5f5e-4b9f-b0b4-1bd345e2afae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t" Oct 29 23:15:00.457315 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.457291 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53b0b99-b590-4aff-aef5-4793abe1f1d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg\" (UID: \"a53b0b99-b590-4aff-aef5-4793abe1f1d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" Oct 29 23:15:00.457413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.457379 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f4756a4-cf22-42c0-89b7-c861adeed9ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh\" (UID: \"5f4756a4-cf22-42c0-89b7-c861adeed9ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh" Oct 29 23:15:00.479415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.479379 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67mnz\" (UniqueName: \"kubernetes.io/projected/a53b0b99-b590-4aff-aef5-4793abe1f1d4-kube-api-access-67mnz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg\" (UID: \"a53b0b99-b590-4aff-aef5-4793abe1f1d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" Oct 29 23:15:00.479415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.479407 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwbw2\" (UniqueName: \"kubernetes.io/projected/5f4756a4-cf22-42c0-89b7-c861adeed9ca-kube-api-access-wwbw2\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh\" (UID: \"5f4756a4-cf22-42c0-89b7-c861adeed9ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh" Oct 29 23:15:00.480723 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.480699 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dnxk\" (UniqueName: \"kubernetes.io/projected/483a170f-cb23-4c9c-9d95-47a7be2344ff-kube-api-access-4dnxk\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw\" (UID: \"483a170f-cb23-4c9c-9d95-47a7be2344ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" Oct 29 23:15:00.480723 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.480707 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvxtf\" (UniqueName: \"kubernetes.io/projected/a89a235c-5f5e-4b9f-b0b4-1bd345e2afae-kube-api-access-kvxtf\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t\" (UID: \"a89a235c-5f5e-4b9f-b0b4-1bd345e2afae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t" Oct 29 23:15:00.485479 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.485456 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" Oct 29 23:15:00.529849 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.529557 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" Oct 29 23:15:00.543663 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.543632 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t" Oct 29 23:15:00.556843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.556799 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87mb\" (UniqueName: \"kubernetes.io/projected/028e4624-bd70-4f79-8026-a998ba9b9f58-kube-api-access-f87mb\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68\" (UID: \"028e4624-bd70-4f79-8026-a998ba9b9f58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" Oct 29 23:15:00.558756 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.558730 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/028e4624-bd70-4f79-8026-a998ba9b9f58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68\" (UID: \"028e4624-bd70-4f79-8026-a998ba9b9f58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" Oct 29 23:15:00.561363 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.560817 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35ad588a-24ca-4823-9d3d-3f281139f7e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm\" (UID: \"35ad588a-24ca-4823-9d3d-3f281139f7e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" Oct 29 23:15:00.563445 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.563423 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4td9f\" (UniqueName: \"kubernetes.io/projected/35ad588a-24ca-4823-9d3d-3f281139f7e6-kube-api-access-4td9f\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm\" (UID: \"35ad588a-24ca-4823-9d3d-3f281139f7e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" Oct 29 23:15:00.563878 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.562214 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35ad588a-24ca-4823-9d3d-3f281139f7e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm\" (UID: \"35ad588a-24ca-4823-9d3d-3f281139f7e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" Oct 29 23:15:00.564401 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.559628 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/028e4624-bd70-4f79-8026-a998ba9b9f58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68\" (UID: \"028e4624-bd70-4f79-8026-a998ba9b9f58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" Oct 29 23:15:00.570710 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.569172 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh" Oct 29 23:15:00.586344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.585624 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f87mb\" (UniqueName: \"kubernetes.io/projected/028e4624-bd70-4f79-8026-a998ba9b9f58-kube-api-access-f87mb\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68\" (UID: \"028e4624-bd70-4f79-8026-a998ba9b9f58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" Oct 29 23:15:00.592683 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.591810 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4td9f\" (UniqueName: \"kubernetes.io/projected/35ad588a-24ca-4823-9d3d-3f281139f7e6-kube-api-access-4td9f\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm\" (UID: \"35ad588a-24ca-4823-9d3d-3f281139f7e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" Oct 29 23:15:00.643529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.636488 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" Oct 29 23:15:00.679759 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.678808 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" Oct 29 23:15:00.836423 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:00.835379 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg"] Oct 29 23:15:01.151421 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:15:01.151301 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483a170f_cb23_4c9c_9d95_47a7be2344ff.slice/crio-8465a497a910f0440f43058303a1ce5bb6dffce0af1cc6e6f0b9d82478d3a865 WatchSource:0}: Error finding container 8465a497a910f0440f43058303a1ce5bb6dffce0af1cc6e6f0b9d82478d3a865: Status 404 returned error can't find the container with id 8465a497a910f0440f43058303a1ce5bb6dffce0af1cc6e6f0b9d82478d3a865 Oct 29 23:15:01.155803 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:15:01.155295 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda89a235c_5f5e_4b9f_b0b4_1bd345e2afae.slice/crio-12b36e30459b4a753043839ed2b468664e9f4c2c7fce007433d1cf153a262be1 WatchSource:0}: Error finding container 12b36e30459b4a753043839ed2b468664e9f4c2c7fce007433d1cf153a262be1: Status 404 returned error can't find the container with id 12b36e30459b4a753043839ed2b468664e9f4c2c7fce007433d1cf153a262be1 Oct 29 23:15:01.158177 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:01.156568 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw"] Oct 29 23:15:01.164014 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:01.163992 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh"] Oct 29 23:15:01.170915 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:01.170885 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t"] Oct 29 23:15:01.214551 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:15:01.214524 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ad588a_24ca_4823_9d3d_3f281139f7e6.slice/crio-c2a5246081e38734ae311c30018bb4476e4d401b1f6b5d3faff7e62433a201d1 WatchSource:0}: Error finding container c2a5246081e38734ae311c30018bb4476e4d401b1f6b5d3faff7e62433a201d1: Status 404 returned error can't find the container with id c2a5246081e38734ae311c30018bb4476e4d401b1f6b5d3faff7e62433a201d1 Oct 29 23:15:01.215118 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:01.215068 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm"] Oct 29 23:15:01.237539 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:01.237508 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68"] Oct 29 23:15:01.485458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:01.485420 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t" event={"ID":"a89a235c-5f5e-4b9f-b0b4-1bd345e2afae","Type":"ContainerStarted","Data":"12b36e30459b4a753043839ed2b468664e9f4c2c7fce007433d1cf153a262be1"} Oct 29 23:15:01.486617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:01.486584 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" event={"ID":"a53b0b99-b590-4aff-aef5-4793abe1f1d4","Type":"ContainerStarted","Data":"c750962935b0b15ca7a2e8c59f999e5eb3bde217a8c85e3d81f4a47ba0e7b195"} Oct 29 23:15:01.486767 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:01.486636 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" event={"ID":"a53b0b99-b590-4aff-aef5-4793abe1f1d4","Type":"ContainerStarted","Data":"10e5a33fd1bf0594fd1024db1aa1b043f422360fd7433ab087aff94382701a25"} Oct 29 23:15:01.487868 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:01.487843 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" event={"ID":"028e4624-bd70-4f79-8026-a998ba9b9f58","Type":"ContainerStarted","Data":"e18bc8a27cfe0e235d624bc7efbb345c1ea89c83df1e36463ef0831568abc210"} Oct 29 23:15:01.489096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:01.489071 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh" event={"ID":"5f4756a4-cf22-42c0-89b7-c861adeed9ca","Type":"ContainerStarted","Data":"57a6eb664a7df39b91b352ee5d37b78db67c11363632f8edc596b3d948f68a97"} Oct 29 23:15:01.490368 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:01.490345 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" event={"ID":"35ad588a-24ca-4823-9d3d-3f281139f7e6","Type":"ContainerStarted","Data":"c2a5246081e38734ae311c30018bb4476e4d401b1f6b5d3faff7e62433a201d1"} Oct 29 23:15:01.491929 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:01.491904 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" event={"ID":"483a170f-cb23-4c9c-9d95-47a7be2344ff","Type":"ContainerStarted","Data":"8465a497a910f0440f43058303a1ce5bb6dffce0af1cc6e6f0b9d82478d3a865"} Oct 29 23:15:01.533918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:01.533874 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" podStartSLOduration=1.533855869 podStartE2EDuration="1.533855869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:01.531922184 +0000 UTC m=+1948.183420717" watchObservedRunningTime="2026-10-29 23:15:01.533855869 +0000 UTC m=+1948.185354389" Oct 29 23:15:02.728210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:02.722951 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" event={"ID":"35ad588a-24ca-4823-9d3d-3f281139f7e6","Type":"ContainerStarted","Data":"8fb0cdedbf5dc50c7e9992df77e882d3fc5c0dd6b94e7003136ba50671bc9cb0"} Oct 29 23:15:02.797224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:02.775750 5057 generic.go:356] "Generic (PLEG): container finished" podID="483a170f-cb23-4c9c-9d95-47a7be2344ff" containerID="6a07a1949bf86f855db453b45e8273ae2c53b8d6a22659d2ff5d017166ee87bf" exitCode=60 Oct 29 23:15:02.797224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:02.775985 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" event={"ID":"483a170f-cb23-4c9c-9d95-47a7be2344ff","Type":"ContainerDied","Data":"6a07a1949bf86f855db453b45e8273ae2c53b8d6a22659d2ff5d017166ee87bf"} Oct 29 23:15:02.797224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:02.776550 5057 scope.go:122] "RemoveContainer" containerID="6a07a1949bf86f855db453b45e8273ae2c53b8d6a22659d2ff5d017166ee87bf" Oct 29 23:15:02.868853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:02.867293 5057 generic.go:356] "Generic (PLEG): container finished" podID="a89a235c-5f5e-4b9f-b0b4-1bd345e2afae" containerID="7ba32631749d64ea73b2685373f095ae5005f59169d574f60d9252ed3ac91eca" exitCode=60 Oct 29 23:15:02.868853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:02.867520 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t" event={"ID":"a89a235c-5f5e-4b9f-b0b4-1bd345e2afae","Type":"ContainerDied","Data":"7ba32631749d64ea73b2685373f095ae5005f59169d574f60d9252ed3ac91eca"} Oct 29 23:15:02.868853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:02.868350 5057 scope.go:122] "RemoveContainer" containerID="7ba32631749d64ea73b2685373f095ae5005f59169d574f60d9252ed3ac91eca" Oct 29 23:15:02.876822 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:02.873857 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" podStartSLOduration=2.873793788 podStartE2EDuration="2.873793788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:02.848849943 +0000 UTC m=+1949.500348475" watchObservedRunningTime="2026-10-29 23:15:02.873793788 +0000 UTC m=+1949.525292320" Oct 29 23:15:02.877529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:02.877503 5057 generic.go:356] "Generic (PLEG): container finished" podID="a53b0b99-b590-4aff-aef5-4793abe1f1d4" containerID="c750962935b0b15ca7a2e8c59f999e5eb3bde217a8c85e3d81f4a47ba0e7b195" exitCode=60 Oct 29 23:15:02.877695 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:02.877675 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" event={"ID":"a53b0b99-b590-4aff-aef5-4793abe1f1d4","Type":"ContainerDied","Data":"c750962935b0b15ca7a2e8c59f999e5eb3bde217a8c85e3d81f4a47ba0e7b195"} Oct 29 23:15:02.878117 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:02.878098 5057 scope.go:122] "RemoveContainer" containerID="c750962935b0b15ca7a2e8c59f999e5eb3bde217a8c85e3d81f4a47ba0e7b195" Oct 29 23:15:02.891871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:02.891841 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" event={"ID":"028e4624-bd70-4f79-8026-a998ba9b9f58","Type":"ContainerStarted","Data":"40bce854b982624e6ccbc528c272eb6f0654769f7dc38bd7163395423bffa17c"} Oct 29 23:15:02.897207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:02.897151 5057 scope.go:122] "RemoveContainer" containerID="40bce854b982624e6ccbc528c272eb6f0654769f7dc38bd7163395423bffa17c" Oct 29 23:15:02.901759 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:02.901711 5057 generic.go:356] "Generic (PLEG): container finished" podID="5f4756a4-cf22-42c0-89b7-c861adeed9ca" containerID="f8e683a12e99b309e2e953459c789798734b9f63437ef20aa0253845e2ce0e9a" exitCode=60 Oct 29 23:15:02.901877 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:02.901767 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh" event={"ID":"5f4756a4-cf22-42c0-89b7-c861adeed9ca","Type":"ContainerDied","Data":"f8e683a12e99b309e2e953459c789798734b9f63437ef20aa0253845e2ce0e9a"} Oct 29 23:15:02.904429 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:02.904248 5057 scope.go:122] "RemoveContainer" containerID="f8e683a12e99b309e2e953459c789798734b9f63437ef20aa0253845e2ce0e9a" Oct 29 23:15:03.914314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:03.914210 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t" event={"ID":"a89a235c-5f5e-4b9f-b0b4-1bd345e2afae","Type":"ContainerStarted","Data":"f08136412dae55f4bf49f207e47ed955fe1740bed9644f75094e18cc3a805112"} Oct 29 23:15:03.918450 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:03.918388 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" event={"ID":"a53b0b99-b590-4aff-aef5-4793abe1f1d4","Type":"ContainerStarted","Data":"8fd2f103dab3581ddd0088f1276e5ac587f1897e1fb0f02aa736d027b0777fce"} Oct 29 23:15:03.920994 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:03.920703 5057 generic.go:356] "Generic (PLEG): container finished" podID="028e4624-bd70-4f79-8026-a998ba9b9f58" containerID="40bce854b982624e6ccbc528c272eb6f0654769f7dc38bd7163395423bffa17c" exitCode=60 Oct 29 23:15:03.920994 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:03.920779 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" event={"ID":"028e4624-bd70-4f79-8026-a998ba9b9f58","Type":"ContainerDied","Data":"40bce854b982624e6ccbc528c272eb6f0654769f7dc38bd7163395423bffa17c"} Oct 29 23:15:03.920994 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:03.920812 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" event={"ID":"028e4624-bd70-4f79-8026-a998ba9b9f58","Type":"ContainerStarted","Data":"58e8b004b444ff25159190c3983528c6bc0932a505b81c8948ec35d7fdc93cc3"} Oct 29 23:15:03.926603 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:03.926576 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh" event={"ID":"5f4756a4-cf22-42c0-89b7-c861adeed9ca","Type":"ContainerStarted","Data":"cd6d06ee091ecb8d9ba78485d17b9f9017d6a22e5a7e91eab87f815c146bf7aa"} Oct 29 23:15:03.929871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:03.929844 5057 generic.go:356] "Generic (PLEG): container finished" podID="35ad588a-24ca-4823-9d3d-3f281139f7e6" containerID="8fb0cdedbf5dc50c7e9992df77e882d3fc5c0dd6b94e7003136ba50671bc9cb0" exitCode=60 Oct 29 23:15:03.929961 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:03.929937 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" event={"ID":"35ad588a-24ca-4823-9d3d-3f281139f7e6","Type":"ContainerDied","Data":"8fb0cdedbf5dc50c7e9992df77e882d3fc5c0dd6b94e7003136ba50671bc9cb0"} Oct 29 23:15:03.930288 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:03.930241 5057 scope.go:122] "RemoveContainer" containerID="8fb0cdedbf5dc50c7e9992df77e882d3fc5c0dd6b94e7003136ba50671bc9cb0" Oct 29 23:15:03.933085 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:03.932841 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" event={"ID":"483a170f-cb23-4c9c-9d95-47a7be2344ff","Type":"ContainerStarted","Data":"cf4802ff847a8545c1f3e978a336b15d34b95670d6a7dfd56d46a73e83edb73b"} Oct 29 23:15:03.941741 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:15:03.941713 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:15:03.959906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:03.959864 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t" podStartSLOduration=3.959850215 podStartE2EDuration="3.959850215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.957594297 +0000 UTC m=+1950.609092828" watchObservedRunningTime="2026-10-29 23:15:03.959850215 +0000 UTC m=+1950.611348745" Oct 29 23:15:04.010055 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.010006 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" podStartSLOduration=4.009993029 podStartE2EDuration="4.009993029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:04.00935774 +0000 UTC m=+1950.660856269" watchObservedRunningTime="2026-10-29 23:15:04.009993029 +0000 UTC m=+1950.661491559" Oct 29 23:15:04.063176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.063111 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" podStartSLOduration=4.063093078 podStartE2EDuration="4.063093078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:04.056247255 +0000 UTC m=+1950.707745785" watchObservedRunningTime="2026-10-29 23:15:04.063093078 +0000 UTC m=+1950.714591609" Oct 29 23:15:04.187714 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.187652 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh" podStartSLOduration=4.187634158 podStartE2EDuration="4.187634158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:04.184790326 +0000 UTC m=+1950.836288857" watchObservedRunningTime="2026-10-29 23:15:04.187634158 +0000 UTC m=+1950.839132694" Oct 29 23:15:04.946389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.946351 5057 generic.go:356] "Generic (PLEG): container finished" podID="a89a235c-5f5e-4b9f-b0b4-1bd345e2afae" containerID="f08136412dae55f4bf49f207e47ed955fe1740bed9644f75094e18cc3a805112" exitCode=60 Oct 29 23:15:04.947016 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.946467 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t" event={"ID":"a89a235c-5f5e-4b9f-b0b4-1bd345e2afae","Type":"ContainerDied","Data":"f08136412dae55f4bf49f207e47ed955fe1740bed9644f75094e18cc3a805112"} Oct 29 23:15:04.947016 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.946528 5057 scope.go:122] "RemoveContainer" containerID="7ba32631749d64ea73b2685373f095ae5005f59169d574f60d9252ed3ac91eca" Oct 29 23:15:04.947141 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.947126 5057 scope.go:122] "RemoveContainer" containerID="f08136412dae55f4bf49f207e47ed955fe1740bed9644f75094e18cc3a805112" Oct 29 23:15:04.947640 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:15:04.947612 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t_odh-ai-gateway-infra(a89a235c-5f5e-4b9f-b0b4-1bd345e2afae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t" podUID="a89a235c-5f5e-4b9f-b0b4-1bd345e2afae" Oct 29 23:15:04.949353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.949312 5057 generic.go:356] "Generic (PLEG): container finished" podID="a53b0b99-b590-4aff-aef5-4793abe1f1d4" containerID="8fd2f103dab3581ddd0088f1276e5ac587f1897e1fb0f02aa736d027b0777fce" exitCode=60 Oct 29 23:15:04.949466 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.949400 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" event={"ID":"a53b0b99-b590-4aff-aef5-4793abe1f1d4","Type":"ContainerDied","Data":"8fd2f103dab3581ddd0088f1276e5ac587f1897e1fb0f02aa736d027b0777fce"} Oct 29 23:15:04.949857 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.949821 5057 scope.go:122] "RemoveContainer" containerID="8fd2f103dab3581ddd0088f1276e5ac587f1897e1fb0f02aa736d027b0777fce" Oct 29 23:15:04.950318 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:15:04.950287 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg_odh-ai-gateway-infra(a53b0b99-b590-4aff-aef5-4793abe1f1d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" podUID="a53b0b99-b590-4aff-aef5-4793abe1f1d4" Oct 29 23:15:04.951709 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.951684 5057 generic.go:356] "Generic (PLEG): container finished" podID="028e4624-bd70-4f79-8026-a998ba9b9f58" containerID="58e8b004b444ff25159190c3983528c6bc0932a505b81c8948ec35d7fdc93cc3" exitCode=60 Oct 29 23:15:04.952246 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.951779 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" event={"ID":"028e4624-bd70-4f79-8026-a998ba9b9f58","Type":"ContainerDied","Data":"58e8b004b444ff25159190c3983528c6bc0932a505b81c8948ec35d7fdc93cc3"} Oct 29 23:15:04.952246 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.952198 5057 scope.go:122] "RemoveContainer" containerID="58e8b004b444ff25159190c3983528c6bc0932a505b81c8948ec35d7fdc93cc3" Oct 29 23:15:04.952719 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:15:04.952692 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68_odh-ai-gateway-infra(028e4624-bd70-4f79-8026-a998ba9b9f58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" podUID="028e4624-bd70-4f79-8026-a998ba9b9f58" Oct 29 23:15:04.954684 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.954660 5057 generic.go:356] "Generic (PLEG): container finished" podID="5f4756a4-cf22-42c0-89b7-c861adeed9ca" containerID="cd6d06ee091ecb8d9ba78485d17b9f9017d6a22e5a7e91eab87f815c146bf7aa" exitCode=60 Oct 29 23:15:04.954788 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.954735 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh" event={"ID":"5f4756a4-cf22-42c0-89b7-c861adeed9ca","Type":"ContainerDied","Data":"cd6d06ee091ecb8d9ba78485d17b9f9017d6a22e5a7e91eab87f815c146bf7aa"} Oct 29 23:15:04.955100 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.955082 5057 scope.go:122] "RemoveContainer" containerID="cd6d06ee091ecb8d9ba78485d17b9f9017d6a22e5a7e91eab87f815c146bf7aa" Oct 29 23:15:04.955510 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:15:04.955487 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh_odh-ai-gateway-infra(5f4756a4-cf22-42c0-89b7-c861adeed9ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh" podUID="5f4756a4-cf22-42c0-89b7-c861adeed9ca" Oct 29 23:15:04.958061 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.958038 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" event={"ID":"35ad588a-24ca-4823-9d3d-3f281139f7e6","Type":"ContainerStarted","Data":"dc2781e1a10c58cfa869b0250d84617b6d486346bd956651f78197cbeaf10674"} Oct 29 23:15:04.960241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.960217 5057 generic.go:356] "Generic (PLEG): container finished" podID="483a170f-cb23-4c9c-9d95-47a7be2344ff" containerID="cf4802ff847a8545c1f3e978a336b15d34b95670d6a7dfd56d46a73e83edb73b" exitCode=60 Oct 29 23:15:04.960357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.960285 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" event={"ID":"483a170f-cb23-4c9c-9d95-47a7be2344ff","Type":"ContainerDied","Data":"cf4802ff847a8545c1f3e978a336b15d34b95670d6a7dfd56d46a73e83edb73b"} Oct 29 23:15:04.960547 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.960531 5057 scope.go:122] "RemoveContainer" containerID="cf4802ff847a8545c1f3e978a336b15d34b95670d6a7dfd56d46a73e83edb73b" Oct 29 23:15:04.960835 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:15:04.960815 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw_odh-ai-gateway-infra(483a170f-cb23-4c9c-9d95-47a7be2344ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" podUID="483a170f-cb23-4c9c-9d95-47a7be2344ff" Oct 29 23:15:04.962494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.962469 5057 scope.go:122] "RemoveContainer" containerID="c750962935b0b15ca7a2e8c59f999e5eb3bde217a8c85e3d81f4a47ba0e7b195" Oct 29 23:15:04.978031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.978009 5057 scope.go:122] "RemoveContainer" containerID="40bce854b982624e6ccbc528c272eb6f0654769f7dc38bd7163395423bffa17c" Oct 29 23:15:04.995685 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:04.995661 5057 scope.go:122] "RemoveContainer" containerID="f8e683a12e99b309e2e953459c789798734b9f63437ef20aa0253845e2ce0e9a" Oct 29 23:15:05.010472 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:05.010449 5057 scope.go:122] "RemoveContainer" containerID="6a07a1949bf86f855db453b45e8273ae2c53b8d6a22659d2ff5d017166ee87bf" Oct 29 23:15:05.973029 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:05.972996 5057 scope.go:122] "RemoveContainer" containerID="f08136412dae55f4bf49f207e47ed955fe1740bed9644f75094e18cc3a805112" Oct 29 23:15:05.973557 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:15:05.973462 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t_odh-ai-gateway-infra(a89a235c-5f5e-4b9f-b0b4-1bd345e2afae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t" podUID="a89a235c-5f5e-4b9f-b0b4-1bd345e2afae" Oct 29 23:15:05.974717 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:05.974693 5057 scope.go:122] "RemoveContainer" containerID="8fd2f103dab3581ddd0088f1276e5ac587f1897e1fb0f02aa736d027b0777fce" Oct 29 23:15:05.975097 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:15:05.975073 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg_odh-ai-gateway-infra(a53b0b99-b590-4aff-aef5-4793abe1f1d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" podUID="a53b0b99-b590-4aff-aef5-4793abe1f1d4" Oct 29 23:15:05.976491 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:05.976469 5057 scope.go:122] "RemoveContainer" containerID="58e8b004b444ff25159190c3983528c6bc0932a505b81c8948ec35d7fdc93cc3" Oct 29 23:15:05.976834 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:15:05.976807 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68_odh-ai-gateway-infra(028e4624-bd70-4f79-8026-a998ba9b9f58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" podUID="028e4624-bd70-4f79-8026-a998ba9b9f58" Oct 29 23:15:05.978170 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:05.978152 5057 scope.go:122] "RemoveContainer" containerID="cd6d06ee091ecb8d9ba78485d17b9f9017d6a22e5a7e91eab87f815c146bf7aa" Oct 29 23:15:05.978451 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:15:05.978428 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh_odh-ai-gateway-infra(5f4756a4-cf22-42c0-89b7-c861adeed9ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh" podUID="5f4756a4-cf22-42c0-89b7-c861adeed9ca" Oct 29 23:15:05.979719 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:05.979700 5057 generic.go:356] "Generic (PLEG): container finished" podID="35ad588a-24ca-4823-9d3d-3f281139f7e6" containerID="dc2781e1a10c58cfa869b0250d84617b6d486346bd956651f78197cbeaf10674" exitCode=60 Oct 29 23:15:05.979807 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:05.979763 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" event={"ID":"35ad588a-24ca-4823-9d3d-3f281139f7e6","Type":"ContainerDied","Data":"dc2781e1a10c58cfa869b0250d84617b6d486346bd956651f78197cbeaf10674"} Oct 29 23:15:05.979850 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:05.979806 5057 scope.go:122] "RemoveContainer" containerID="8fb0cdedbf5dc50c7e9992df77e882d3fc5c0dd6b94e7003136ba50671bc9cb0" Oct 29 23:15:05.980056 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:05.980035 5057 scope.go:122] "RemoveContainer" containerID="dc2781e1a10c58cfa869b0250d84617b6d486346bd956651f78197cbeaf10674" Oct 29 23:15:05.980431 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:15:05.980346 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm_odh-ai-gateway-infra(35ad588a-24ca-4823-9d3d-3f281139f7e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" podUID="35ad588a-24ca-4823-9d3d-3f281139f7e6" Oct 29 23:15:05.981698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:05.981680 5057 scope.go:122] "RemoveContainer" containerID="cf4802ff847a8545c1f3e978a336b15d34b95670d6a7dfd56d46a73e83edb73b" Oct 29 23:15:05.981899 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:15:05.981882 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw_odh-ai-gateway-infra(483a170f-cb23-4c9c-9d95-47a7be2344ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" podUID="483a170f-cb23-4c9c-9d95-47a7be2344ff" Oct 29 23:15:06.991906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:06.991878 5057 scope.go:122] "RemoveContainer" containerID="dc2781e1a10c58cfa869b0250d84617b6d486346bd956651f78197cbeaf10674" Oct 29 23:15:06.992357 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:15:06.992300 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm_odh-ai-gateway-infra(35ad588a-24ca-4823-9d3d-3f281139f7e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" podUID="35ad588a-24ca-4823-9d3d-3f281139f7e6" Oct 29 23:15:13.136015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:13.135963 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:15:15.492212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:15.487033 5057 scope.go:122] "RemoveContainer" containerID="8fd2f103dab3581ddd0088f1276e5ac587f1897e1fb0f02aa736d027b0777fce" Oct 29 23:15:15.500209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:15.497998 5057 scope.go:122] "RemoveContainer" containerID="f08136412dae55f4bf49f207e47ed955fe1740bed9644f75094e18cc3a805112" Oct 29 23:15:16.189979 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:16.189947 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t" event={"ID":"a89a235c-5f5e-4b9f-b0b4-1bd345e2afae","Type":"ContainerStarted","Data":"3ff7ca364e5247e31e21c3380673b575713d68e717277a00a4cda1fe8364a35b"} Oct 29 23:15:16.192658 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:16.192626 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" event={"ID":"a53b0b99-b590-4aff-aef5-4793abe1f1d4","Type":"ContainerStarted","Data":"a89336b2f6c4694e6697724f73d3aa690a026c654e1eaa9a84232cfdb1f35f25"} Oct 29 23:15:16.473426 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:16.473396 5057 scope.go:122] "RemoveContainer" containerID="dc2781e1a10c58cfa869b0250d84617b6d486346bd956651f78197cbeaf10674" Oct 29 23:15:16.473618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:16.473524 5057 scope.go:122] "RemoveContainer" containerID="cd6d06ee091ecb8d9ba78485d17b9f9017d6a22e5a7e91eab87f815c146bf7aa" Oct 29 23:15:17.253269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:17.251276 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t"] Oct 29 23:15:17.301483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:17.301432 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" event={"ID":"35ad588a-24ca-4823-9d3d-3f281139f7e6","Type":"ContainerStarted","Data":"05c4488ecd4532ca58d2ab07ee03a223ce3295d8d581060b1b97a91f3cae0cae"} Oct 29 23:15:17.307904 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:17.307870 5057 generic.go:356] "Generic (PLEG): container finished" podID="a89a235c-5f5e-4b9f-b0b4-1bd345e2afae" containerID="3ff7ca364e5247e31e21c3380673b575713d68e717277a00a4cda1fe8364a35b" exitCode=60 Oct 29 23:15:17.308100 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:17.307990 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t" event={"ID":"a89a235c-5f5e-4b9f-b0b4-1bd345e2afae","Type":"ContainerDied","Data":"3ff7ca364e5247e31e21c3380673b575713d68e717277a00a4cda1fe8364a35b"} Oct 29 23:15:17.310106 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:17.309106 5057 scope.go:122] "RemoveContainer" containerID="f08136412dae55f4bf49f207e47ed955fe1740bed9644f75094e18cc3a805112" Oct 29 23:15:17.315278 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:17.315253 5057 generic.go:356] "Generic (PLEG): container finished" podID="a53b0b99-b590-4aff-aef5-4793abe1f1d4" containerID="a89336b2f6c4694e6697724f73d3aa690a026c654e1eaa9a84232cfdb1f35f25" exitCode=60 Oct 29 23:15:17.315556 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:17.315527 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" event={"ID":"a53b0b99-b590-4aff-aef5-4793abe1f1d4","Type":"ContainerDied","Data":"a89336b2f6c4694e6697724f73d3aa690a026c654e1eaa9a84232cfdb1f35f25"} Oct 29 23:15:17.316971 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:17.316901 5057 scope.go:122] "RemoveContainer" containerID="a89336b2f6c4694e6697724f73d3aa690a026c654e1eaa9a84232cfdb1f35f25" Oct 29 23:15:17.319497 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:15:17.319434 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg_odh-ai-gateway-infra(a53b0b99-b590-4aff-aef5-4793abe1f1d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" podUID="a53b0b99-b590-4aff-aef5-4793abe1f1d4" Oct 29 23:15:17.320013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:17.319936 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg"] Oct 29 23:15:17.344291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:17.343918 5057 scope.go:122] "RemoveContainer" containerID="8fd2f103dab3581ddd0088f1276e5ac587f1897e1fb0f02aa736d027b0777fce" Oct 29 23:15:17.670829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:17.670751 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t" Oct 29 23:15:17.694379 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:17.694349 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a89a235c-5f5e-4b9f-b0b4-1bd345e2afae-kube-api-access-kvxtf\" (UniqueName: \"kubernetes.io/projected/a89a235c-5f5e-4b9f-b0b4-1bd345e2afae-kube-api-access-kvxtf\") pod \"a89a235c-5f5e-4b9f-b0b4-1bd345e2afae\" (UID: \"a89a235c-5f5e-4b9f-b0b4-1bd345e2afae\") " Oct 29 23:15:17.694814 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:17.694794 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a89a235c-5f5e-4b9f-b0b4-1bd345e2afae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a89a235c-5f5e-4b9f-b0b4-1bd345e2afae-maas-api-service-ca\") pod \"a89a235c-5f5e-4b9f-b0b4-1bd345e2afae\" (UID: \"a89a235c-5f5e-4b9f-b0b4-1bd345e2afae\") " Oct 29 23:15:17.695268 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:17.695245 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a89a235c-5f5e-4b9f-b0b4-1bd345e2afae-maas-api-service-ca" pod "a89a235c-5f5e-4b9f-b0b4-1bd345e2afae" (UID: "a89a235c-5f5e-4b9f-b0b4-1bd345e2afae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:17.695636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:17.695613 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a89a235c-5f5e-4b9f-b0b4-1bd345e2afae-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:15:17.697792 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:17.697770 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89a235c-5f5e-4b9f-b0b4-1bd345e2afae-kube-api-access-kvxtf" pod "a89a235c-5f5e-4b9f-b0b4-1bd345e2afae" (UID: "a89a235c-5f5e-4b9f-b0b4-1bd345e2afae"). InnerVolumeSpecName "kube-api-access-kvxtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:17.823266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:17.821901 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvxtf\" (UniqueName: \"kubernetes.io/projected/a89a235c-5f5e-4b9f-b0b4-1bd345e2afae-kube-api-access-kvxtf\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:15:18.352968 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.352928 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t" event={"ID":"a89a235c-5f5e-4b9f-b0b4-1bd345e2afae","Type":"ContainerDied","Data":"12b36e30459b4a753043839ed2b468664e9f4c2c7fce007433d1cf153a262be1"} Oct 29 23:15:18.353666 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.353642 5057 scope.go:122] "RemoveContainer" containerID="3ff7ca364e5247e31e21c3380673b575713d68e717277a00a4cda1fe8364a35b" Oct 29 23:15:18.353898 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.353345 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t" Oct 29 23:15:18.364466 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.363419 5057 generic.go:356] "Generic (PLEG): container finished" podID="5f4756a4-cf22-42c0-89b7-c861adeed9ca" containerID="5293e513fe01dd66e7a5ef8b30f66964d99a8fcec5bd224209eddb479382997f" exitCode=60 Oct 29 23:15:18.364466 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.363500 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh" event={"ID":"5f4756a4-cf22-42c0-89b7-c861adeed9ca","Type":"ContainerDied","Data":"5293e513fe01dd66e7a5ef8b30f66964d99a8fcec5bd224209eddb479382997f"} Oct 29 23:15:18.364466 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.363897 5057 scope.go:122] "RemoveContainer" containerID="5293e513fe01dd66e7a5ef8b30f66964d99a8fcec5bd224209eddb479382997f" Oct 29 23:15:18.364466 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:15:18.364276 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh_odh-ai-gateway-infra(5f4756a4-cf22-42c0-89b7-c861adeed9ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh" podUID="5f4756a4-cf22-42c0-89b7-c861adeed9ca" Oct 29 23:15:18.369710 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.368477 5057 generic.go:356] "Generic (PLEG): container finished" podID="35ad588a-24ca-4823-9d3d-3f281139f7e6" containerID="05c4488ecd4532ca58d2ab07ee03a223ce3295d8d581060b1b97a91f3cae0cae" exitCode=60 Oct 29 23:15:18.369710 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.368622 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" event={"ID":"35ad588a-24ca-4823-9d3d-3f281139f7e6","Type":"ContainerDied","Data":"05c4488ecd4532ca58d2ab07ee03a223ce3295d8d581060b1b97a91f3cae0cae"} Oct 29 23:15:18.369710 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.369101 5057 scope.go:122] "RemoveContainer" containerID="05c4488ecd4532ca58d2ab07ee03a223ce3295d8d581060b1b97a91f3cae0cae" Oct 29 23:15:18.369710 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:15:18.369548 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm_odh-ai-gateway-infra(35ad588a-24ca-4823-9d3d-3f281139f7e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" podUID="35ad588a-24ca-4823-9d3d-3f281139f7e6" Oct 29 23:15:18.370024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.369785 5057 scope.go:122] "RemoveContainer" containerID="cd6d06ee091ecb8d9ba78485d17b9f9017d6a22e5a7e91eab87f815c146bf7aa" Oct 29 23:15:18.371759 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.371501 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm"] Oct 29 23:15:18.389406 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.389375 5057 scope.go:122] "RemoveContainer" containerID="dc2781e1a10c58cfa869b0250d84617b6d486346bd956651f78197cbeaf10674" Oct 29 23:15:18.490161 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.477853 5057 scope.go:122] "RemoveContainer" containerID="cf4802ff847a8545c1f3e978a336b15d34b95670d6a7dfd56d46a73e83edb73b" Oct 29 23:15:18.490161 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.478438 5057 scope.go:122] "RemoveContainer" containerID="58e8b004b444ff25159190c3983528c6bc0932a505b81c8948ec35d7fdc93cc3" Oct 29 23:15:18.490161 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.485992 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t"] Oct 29 23:15:18.490764 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.490610 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888595-pdx6t"] Oct 29 23:15:18.705093 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.705068 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" Oct 29 23:15:18.785916 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.785884 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53b0b99-b590-4aff-aef5-4793abe1f1d4-kube-api-access-67mnz\" (UniqueName: \"kubernetes.io/projected/a53b0b99-b590-4aff-aef5-4793abe1f1d4-kube-api-access-67mnz\") pod \"a53b0b99-b590-4aff-aef5-4793abe1f1d4\" (UID: \"a53b0b99-b590-4aff-aef5-4793abe1f1d4\") " Oct 29 23:15:18.786341 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.786316 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53b0b99-b590-4aff-aef5-4793abe1f1d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53b0b99-b590-4aff-aef5-4793abe1f1d4-maas-api-service-ca\") pod \"a53b0b99-b590-4aff-aef5-4793abe1f1d4\" (UID: \"a53b0b99-b590-4aff-aef5-4793abe1f1d4\") " Oct 29 23:15:18.787001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.786980 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53b0b99-b590-4aff-aef5-4793abe1f1d4-maas-api-service-ca" pod "a53b0b99-b590-4aff-aef5-4793abe1f1d4" (UID: "a53b0b99-b590-4aff-aef5-4793abe1f1d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:18.787263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.787242 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53b0b99-b590-4aff-aef5-4793abe1f1d4-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:15:18.788980 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.788958 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53b0b99-b590-4aff-aef5-4793abe1f1d4-kube-api-access-67mnz" pod "a53b0b99-b590-4aff-aef5-4793abe1f1d4" (UID: "a53b0b99-b590-4aff-aef5-4793abe1f1d4"). InnerVolumeSpecName "kube-api-access-67mnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:18.889772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:18.889640 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67mnz\" (UniqueName: \"kubernetes.io/projected/a53b0b99-b590-4aff-aef5-4793abe1f1d4-kube-api-access-67mnz\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:15:19.409288 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.409136 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" event={"ID":"028e4624-bd70-4f79-8026-a998ba9b9f58","Type":"ContainerStarted","Data":"86e15be26562d01b429247c39cbbd708d5b549f020fd3ecfb6ca1063378f4bd8"} Oct 29 23:15:19.431166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.431125 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" event={"ID":"483a170f-cb23-4c9c-9d95-47a7be2344ff","Type":"ContainerStarted","Data":"99af8c240908077d3a8f88bdff881695689299b7305ee097459cbe124c32df22"} Oct 29 23:15:19.440210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.440128 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh"] Oct 29 23:15:19.441242 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.441162 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" event={"ID":"a53b0b99-b590-4aff-aef5-4793abe1f1d4","Type":"ContainerDied","Data":"10e5a33fd1bf0594fd1024db1aa1b043f422360fd7433ab087aff94382701a25"} Oct 29 23:15:19.441375 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.441281 5057 scope.go:122] "RemoveContainer" containerID="a89336b2f6c4694e6697724f73d3aa690a026c654e1eaa9a84232cfdb1f35f25" Oct 29 23:15:19.441483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.441463 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg" Oct 29 23:15:19.492344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.490553 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a89a235c-5f5e-4b9f-b0b4-1bd345e2afae" path="/var/lib/kubelet/pods/a89a235c-5f5e-4b9f-b0b4-1bd345e2afae/volumes" Oct 29 23:15:19.672586 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.672512 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg"] Oct 29 23:15:19.685405 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.685373 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888595-c4cgg"] Oct 29 23:15:19.726996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.726964 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" Oct 29 23:15:19.782377 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.779084 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh" Oct 29 23:15:19.831374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.815759 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f4756a4-cf22-42c0-89b7-c861adeed9ca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f4756a4-cf22-42c0-89b7-c861adeed9ca-maas-api-service-ca\") pod \"5f4756a4-cf22-42c0-89b7-c861adeed9ca\" (UID: \"5f4756a4-cf22-42c0-89b7-c861adeed9ca\") " Oct 29 23:15:19.831374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.815930 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f4756a4-cf22-42c0-89b7-c861adeed9ca-kube-api-access-wwbw2\" (UniqueName: \"kubernetes.io/projected/5f4756a4-cf22-42c0-89b7-c861adeed9ca-kube-api-access-wwbw2\") pod \"5f4756a4-cf22-42c0-89b7-c861adeed9ca\" (UID: \"5f4756a4-cf22-42c0-89b7-c861adeed9ca\") " Oct 29 23:15:19.831374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.816015 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35ad588a-24ca-4823-9d3d-3f281139f7e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35ad588a-24ca-4823-9d3d-3f281139f7e6-maas-api-service-ca\") pod \"35ad588a-24ca-4823-9d3d-3f281139f7e6\" (UID: \"35ad588a-24ca-4823-9d3d-3f281139f7e6\") " Oct 29 23:15:19.831374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.816102 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35ad588a-24ca-4823-9d3d-3f281139f7e6-kube-api-access-4td9f\" (UniqueName: \"kubernetes.io/projected/35ad588a-24ca-4823-9d3d-3f281139f7e6-kube-api-access-4td9f\") pod \"35ad588a-24ca-4823-9d3d-3f281139f7e6\" (UID: \"35ad588a-24ca-4823-9d3d-3f281139f7e6\") " Oct 29 23:15:19.831374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.819847 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f4756a4-cf22-42c0-89b7-c861adeed9ca-maas-api-service-ca" pod "5f4756a4-cf22-42c0-89b7-c861adeed9ca" (UID: "5f4756a4-cf22-42c0-89b7-c861adeed9ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:19.831374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.820210 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35ad588a-24ca-4823-9d3d-3f281139f7e6-maas-api-service-ca" pod "35ad588a-24ca-4823-9d3d-3f281139f7e6" (UID: "35ad588a-24ca-4823-9d3d-3f281139f7e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:19.901117 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.877500 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ad588a-24ca-4823-9d3d-3f281139f7e6-kube-api-access-4td9f" pod "35ad588a-24ca-4823-9d3d-3f281139f7e6" (UID: "35ad588a-24ca-4823-9d3d-3f281139f7e6"). InnerVolumeSpecName "kube-api-access-4td9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:19.901117 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.878111 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4756a4-cf22-42c0-89b7-c861adeed9ca-kube-api-access-wwbw2" pod "5f4756a4-cf22-42c0-89b7-c861adeed9ca" (UID: "5f4756a4-cf22-42c0-89b7-c861adeed9ca"). InnerVolumeSpecName "kube-api-access-wwbw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:19.919014 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.918977 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwbw2\" (UniqueName: \"kubernetes.io/projected/5f4756a4-cf22-42c0-89b7-c861adeed9ca-kube-api-access-wwbw2\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:15:19.919247 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.919231 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35ad588a-24ca-4823-9d3d-3f281139f7e6-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:15:19.919338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.919326 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4td9f\" (UniqueName: \"kubernetes.io/projected/35ad588a-24ca-4823-9d3d-3f281139f7e6-kube-api-access-4td9f\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:15:19.919423 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:19.919411 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f4756a4-cf22-42c0-89b7-c861adeed9ca-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:15:20.451704 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:20.451663 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh" event={"ID":"5f4756a4-cf22-42c0-89b7-c861adeed9ca","Type":"ContainerDied","Data":"57a6eb664a7df39b91b352ee5d37b78db67c11363632f8edc596b3d948f68a97"} Oct 29 23:15:20.452169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:20.451789 5057 scope.go:122] "RemoveContainer" containerID="5293e513fe01dd66e7a5ef8b30f66964d99a8fcec5bd224209eddb479382997f" Oct 29 23:15:20.452169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:20.451674 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh" Oct 29 23:15:20.454082 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:20.454057 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" event={"ID":"35ad588a-24ca-4823-9d3d-3f281139f7e6","Type":"ContainerDied","Data":"c2a5246081e38734ae311c30018bb4476e4d401b1f6b5d3faff7e62433a201d1"} Oct 29 23:15:20.454399 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:20.454375 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm" Oct 29 23:15:20.470602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:20.464486 5057 generic.go:356] "Generic (PLEG): container finished" podID="483a170f-cb23-4c9c-9d95-47a7be2344ff" containerID="99af8c240908077d3a8f88bdff881695689299b7305ee097459cbe124c32df22" exitCode=60 Oct 29 23:15:20.470602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:20.464743 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" event={"ID":"483a170f-cb23-4c9c-9d95-47a7be2344ff","Type":"ContainerDied","Data":"99af8c240908077d3a8f88bdff881695689299b7305ee097459cbe124c32df22"} Oct 29 23:15:20.470602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:20.465145 5057 scope.go:122] "RemoveContainer" containerID="99af8c240908077d3a8f88bdff881695689299b7305ee097459cbe124c32df22" Oct 29 23:15:20.470602 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:15:20.465849 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw_odh-ai-gateway-infra(483a170f-cb23-4c9c-9d95-47a7be2344ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" podUID="483a170f-cb23-4c9c-9d95-47a7be2344ff" Oct 29 23:15:20.470602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:20.466114 5057 scope.go:122] "RemoveContainer" containerID="05c4488ecd4532ca58d2ab07ee03a223ce3295d8d581060b1b97a91f3cae0cae" Oct 29 23:15:20.470602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:20.469054 5057 generic.go:356] "Generic (PLEG): container finished" podID="028e4624-bd70-4f79-8026-a998ba9b9f58" containerID="86e15be26562d01b429247c39cbbd708d5b549f020fd3ecfb6ca1063378f4bd8" exitCode=60 Oct 29 23:15:20.470602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:20.469096 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" event={"ID":"028e4624-bd70-4f79-8026-a998ba9b9f58","Type":"ContainerDied","Data":"86e15be26562d01b429247c39cbbd708d5b549f020fd3ecfb6ca1063378f4bd8"} Oct 29 23:15:20.470602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:20.469498 5057 scope.go:122] "RemoveContainer" containerID="86e15be26562d01b429247c39cbbd708d5b549f020fd3ecfb6ca1063378f4bd8" Oct 29 23:15:20.470602 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:15:20.469798 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68_odh-ai-gateway-infra(028e4624-bd70-4f79-8026-a998ba9b9f58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" podUID="028e4624-bd70-4f79-8026-a998ba9b9f58" Oct 29 23:15:20.472981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:20.472958 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68"] Oct 29 23:15:20.479822 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:20.479805 5057 scope.go:122] "RemoveContainer" containerID="cf4802ff847a8545c1f3e978a336b15d34b95670d6a7dfd56d46a73e83edb73b" Oct 29 23:15:20.495339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:20.495171 5057 scope.go:122] "RemoveContainer" containerID="58e8b004b444ff25159190c3983528c6bc0932a505b81c8948ec35d7fdc93cc3" Oct 29 23:15:20.520012 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:20.519983 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw"] Oct 29 23:15:20.524468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:20.524440 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh"] Oct 29 23:15:20.532120 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:20.532091 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888595-km5kh"] Oct 29 23:15:20.697286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:20.688029 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm"] Oct 29 23:15:20.729149 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:20.729047 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888595-l2cvm"] Oct 29 23:15:21.486529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:21.486475 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35ad588a-24ca-4823-9d3d-3f281139f7e6" path="/var/lib/kubelet/pods/35ad588a-24ca-4823-9d3d-3f281139f7e6/volumes" Oct 29 23:15:21.487314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:21.487289 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f4756a4-cf22-42c0-89b7-c861adeed9ca" path="/var/lib/kubelet/pods/5f4756a4-cf22-42c0-89b7-c861adeed9ca/volumes" Oct 29 23:15:21.487911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:21.487888 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a53b0b99-b590-4aff-aef5-4793abe1f1d4" path="/var/lib/kubelet/pods/a53b0b99-b590-4aff-aef5-4793abe1f1d4/volumes" Oct 29 23:15:21.983154 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:21.983129 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" Oct 29 23:15:21.989079 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:21.989051 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" Oct 29 23:15:21.998516 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:21.993387 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/028e4624-bd70-4f79-8026-a998ba9b9f58-kube-api-access-f87mb\" (UniqueName: \"kubernetes.io/projected/028e4624-bd70-4f79-8026-a998ba9b9f58-kube-api-access-f87mb\") pod \"028e4624-bd70-4f79-8026-a998ba9b9f58\" (UID: \"028e4624-bd70-4f79-8026-a998ba9b9f58\") " Oct 29 23:15:21.998516 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:21.993706 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/028e4624-bd70-4f79-8026-a998ba9b9f58-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/028e4624-bd70-4f79-8026-a998ba9b9f58-maas-api-service-ca\") pod \"028e4624-bd70-4f79-8026-a998ba9b9f58\" (UID: \"028e4624-bd70-4f79-8026-a998ba9b9f58\") " Oct 29 23:15:21.998516 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:21.994602 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/028e4624-bd70-4f79-8026-a998ba9b9f58-maas-api-service-ca" pod "028e4624-bd70-4f79-8026-a998ba9b9f58" (UID: "028e4624-bd70-4f79-8026-a998ba9b9f58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:22.001433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:22.001392 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028e4624-bd70-4f79-8026-a998ba9b9f58-kube-api-access-f87mb" pod "028e4624-bd70-4f79-8026-a998ba9b9f58" (UID: "028e4624-bd70-4f79-8026-a998ba9b9f58"). InnerVolumeSpecName "kube-api-access-f87mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:22.095359 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:22.095295 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/483a170f-cb23-4c9c-9d95-47a7be2344ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483a170f-cb23-4c9c-9d95-47a7be2344ff-maas-api-service-ca\") pod \"483a170f-cb23-4c9c-9d95-47a7be2344ff\" (UID: \"483a170f-cb23-4c9c-9d95-47a7be2344ff\") " Oct 29 23:15:22.095616 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:22.095595 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/483a170f-cb23-4c9c-9d95-47a7be2344ff-kube-api-access-4dnxk\" (UniqueName: \"kubernetes.io/projected/483a170f-cb23-4c9c-9d95-47a7be2344ff-kube-api-access-4dnxk\") pod \"483a170f-cb23-4c9c-9d95-47a7be2344ff\" (UID: \"483a170f-cb23-4c9c-9d95-47a7be2344ff\") " Oct 29 23:15:22.096547 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:22.096515 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f87mb\" (UniqueName: \"kubernetes.io/projected/028e4624-bd70-4f79-8026-a998ba9b9f58-kube-api-access-f87mb\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:15:22.096668 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:22.096589 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/028e4624-bd70-4f79-8026-a998ba9b9f58-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:15:22.100938 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:22.100911 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/483a170f-cb23-4c9c-9d95-47a7be2344ff-maas-api-service-ca" pod "483a170f-cb23-4c9c-9d95-47a7be2344ff" (UID: "483a170f-cb23-4c9c-9d95-47a7be2344ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:22.102647 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:22.102617 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/483a170f-cb23-4c9c-9d95-47a7be2344ff-kube-api-access-4dnxk" pod "483a170f-cb23-4c9c-9d95-47a7be2344ff" (UID: "483a170f-cb23-4c9c-9d95-47a7be2344ff"). InnerVolumeSpecName "kube-api-access-4dnxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:22.208110 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:22.208073 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483a170f-cb23-4c9c-9d95-47a7be2344ff-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:15:22.230758 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:22.230526 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dnxk\" (UniqueName: \"kubernetes.io/projected/483a170f-cb23-4c9c-9d95-47a7be2344ff-kube-api-access-4dnxk\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:15:22.499926 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:22.499840 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" Oct 29 23:15:22.499926 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:22.499854 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68" event={"ID":"028e4624-bd70-4f79-8026-a998ba9b9f58","Type":"ContainerDied","Data":"e18bc8a27cfe0e235d624bc7efbb345c1ea89c83df1e36463ef0831568abc210"} Oct 29 23:15:22.499926 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:22.499907 5057 scope.go:122] "RemoveContainer" containerID="86e15be26562d01b429247c39cbbd708d5b549f020fd3ecfb6ca1063378f4bd8" Oct 29 23:15:22.502599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:22.502573 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" event={"ID":"483a170f-cb23-4c9c-9d95-47a7be2344ff","Type":"ContainerDied","Data":"8465a497a910f0440f43058303a1ce5bb6dffce0af1cc6e6f0b9d82478d3a865"} Oct 29 23:15:22.502766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:22.502663 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw" Oct 29 23:15:22.514809 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:22.514786 5057 scope.go:122] "RemoveContainer" containerID="99af8c240908077d3a8f88bdff881695689299b7305ee097459cbe124c32df22" Oct 29 23:15:22.562283 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:22.562251 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw"] Oct 29 23:15:22.569809 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:22.569778 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888595-wzwbw"] Oct 29 23:15:22.609394 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:22.609356 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68"] Oct 29 23:15:22.615606 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:22.615577 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888595-wmh68"] Oct 29 23:15:23.485522 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:23.485486 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="028e4624-bd70-4f79-8026-a998ba9b9f58" path="/var/lib/kubelet/pods/028e4624-bd70-4f79-8026-a998ba9b9f58/volumes" Oct 29 23:15:23.486015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:23.485980 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="483a170f-cb23-4c9c-9d95-47a7be2344ff" path="/var/lib/kubelet/pods/483a170f-cb23-4c9c-9d95-47a7be2344ff/volumes" Oct 29 23:15:25.608776 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:25.608739 5057 generic.go:356] "Generic (PLEG): container finished" podID="408b4a17-ac05-47f5-bbcb-ed398229570c" containerID="0abed984960160f246c3f1ddf058d37cfd85832fe262ba396e07da3a699545ce" exitCode=0 Oct 29 23:15:25.609304 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:25.608818 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" event={"ID":"408b4a17-ac05-47f5-bbcb-ed398229570c","Type":"ContainerDied","Data":"0abed984960160f246c3f1ddf058d37cfd85832fe262ba396e07da3a699545ce"} Oct 29 23:15:25.609429 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:25.609411 5057 scope.go:122] "RemoveContainer" containerID="0abed984960160f246c3f1ddf058d37cfd85832fe262ba396e07da3a699545ce" Oct 29 23:15:25.611503 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:25.611475 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:15:25.611604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:25.611542 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:15:25.611604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:25.611575 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:15:26.620532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:26.620443 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" event={"ID":"408b4a17-ac05-47f5-bbcb-ed398229570c","Type":"ContainerStarted","Data":"276620c57c73fdf3f28ccb85e9fabbf67a5d7b69bfee33ca4d81a2f01fe4a955"} Oct 29 23:15:27.593614 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:27.593585 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:15:27.597220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:27.595409 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 29 23:15:27.597220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:27.595470 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" podUID="408b4a17-ac05-47f5-bbcb-ed398229570c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 29 23:15:28.590650 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:28.590616 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 29 23:15:28.591049 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:28.590667 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" podUID="408b4a17-ac05-47f5-bbcb-ed398229570c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 29 23:15:28.698942 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:28.696416 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:15:29.590354 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:29.590322 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 29 23:15:29.590538 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:29.590375 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" podUID="408b4a17-ac05-47f5-bbcb-ed398229570c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 29 23:15:30.590864 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:30.590828 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 29 23:15:30.591350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:30.590882 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" podUID="408b4a17-ac05-47f5-bbcb-ed398229570c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 29 23:15:31.591594 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:31.591540 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 29 23:15:31.591995 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:31.591610 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" podUID="408b4a17-ac05-47f5-bbcb-ed398229570c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 29 23:15:32.595508 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:32.595478 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:15:32.596664 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:32.596640 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:15:33.916359 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:15:33.916314 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:15:40.364036 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:40.363994 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:15:50.777618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.775826 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5"] Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.785810 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a53b0b99-b590-4aff-aef5-4793abe1f1d4" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.785841 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53b0b99-b590-4aff-aef5-4793abe1f1d4" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.785857 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5f4756a4-cf22-42c0-89b7-c861adeed9ca" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.785865 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f4756a4-cf22-42c0-89b7-c861adeed9ca" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.785878 5057 cpu_manager.go:414] "containerMap: removing container" podUID="483a170f-cb23-4c9c-9d95-47a7be2344ff" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.785884 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="483a170f-cb23-4c9c-9d95-47a7be2344ff" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.785898 5057 cpu_manager.go:414] "containerMap: removing container" podUID="028e4624-bd70-4f79-8026-a998ba9b9f58" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.785907 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="028e4624-bd70-4f79-8026-a998ba9b9f58" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.785917 5057 cpu_manager.go:414] "containerMap: removing container" podUID="35ad588a-24ca-4823-9d3d-3f281139f7e6" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.785923 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35ad588a-24ca-4823-9d3d-3f281139f7e6" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.785935 5057 cpu_manager.go:414] "containerMap: removing container" podUID="483a170f-cb23-4c9c-9d95-47a7be2344ff" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.785942 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="483a170f-cb23-4c9c-9d95-47a7be2344ff" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.785956 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a89a235c-5f5e-4b9f-b0b4-1bd345e2afae" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.785963 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a89a235c-5f5e-4b9f-b0b4-1bd345e2afae" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.785986 5057 cpu_manager.go:414] "containerMap: removing container" podUID="35ad588a-24ca-4823-9d3d-3f281139f7e6" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.785994 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35ad588a-24ca-4823-9d3d-3f281139f7e6" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786006 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a53b0b99-b590-4aff-aef5-4793abe1f1d4" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786013 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53b0b99-b590-4aff-aef5-4793abe1f1d4" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786022 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a89a235c-5f5e-4b9f-b0b4-1bd345e2afae" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786029 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a89a235c-5f5e-4b9f-b0b4-1bd345e2afae" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786083 5057 cpu_manager.go:414] "containerMap: removing container" podUID="483a170f-cb23-4c9c-9d95-47a7be2344ff" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786091 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="483a170f-cb23-4c9c-9d95-47a7be2344ff" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786124 5057 cpu_manager.go:414] "containerMap: removing container" podUID="028e4624-bd70-4f79-8026-a998ba9b9f58" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786131 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="028e4624-bd70-4f79-8026-a998ba9b9f58" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786153 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5f4756a4-cf22-42c0-89b7-c861adeed9ca" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786160 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f4756a4-cf22-42c0-89b7-c861adeed9ca" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786416 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a89a235c-5f5e-4b9f-b0b4-1bd345e2afae" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786442 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="35ad588a-24ca-4823-9d3d-3f281139f7e6" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786453 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53b0b99-b590-4aff-aef5-4793abe1f1d4" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786463 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="028e4624-bd70-4f79-8026-a998ba9b9f58" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786474 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="483a170f-cb23-4c9c-9d95-47a7be2344ff" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786485 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f4756a4-cf22-42c0-89b7-c861adeed9ca" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786496 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="028e4624-bd70-4f79-8026-a998ba9b9f58" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786509 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f4756a4-cf22-42c0-89b7-c861adeed9ca" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786520 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="028e4624-bd70-4f79-8026-a998ba9b9f58" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786528 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="35ad588a-24ca-4823-9d3d-3f281139f7e6" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786537 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="483a170f-cb23-4c9c-9d95-47a7be2344ff" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786558 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53b0b99-b590-4aff-aef5-4793abe1f1d4" containerName="cleanup" Oct 29 23:15:50.786770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.786566 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a89a235c-5f5e-4b9f-b0b4-1bd345e2afae" containerName="cleanup" Oct 29 23:15:50.793843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.793813 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.794092 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.793819 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5"] Oct 29 23:15:50.811063 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.811036 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-2591f6-openshift-default-dockercfg-5rxmx\"" Oct 29 23:15:50.877718 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.877675 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-token\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.877927 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.877726 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-workload-socket\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.877927 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.877754 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-podinfo\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.877927 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.877892 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-credential-socket\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.878108 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.878024 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc22e58-ddce-4866-b17d-8672d775fc4a-istiod-ca-cert\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.878177 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.878105 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-envoy\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.878177 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.878161 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrnnc\" (UniqueName: \"kubernetes.io/projected/4cc22e58-ddce-4866-b17d-8672d775fc4a-kube-api-access-jrnnc\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.878356 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.878333 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-data\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.878490 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.878383 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-workload-certs\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.980259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.980138 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrnnc\" (UniqueName: \"kubernetes.io/projected/4cc22e58-ddce-4866-b17d-8672d775fc4a-kube-api-access-jrnnc\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.980510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.980301 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-data\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.980510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.980352 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-workload-certs\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.980658 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.980545 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-token\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.980658 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.980590 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-workload-socket\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.980658 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.980616 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-podinfo\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.980819 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.980678 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-credential-socket\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.980819 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.980798 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc22e58-ddce-4866-b17d-8672d775fc4a-istiod-ca-cert\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.980941 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.980879 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-envoy\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.981998 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.981764 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-credential-socket\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.981998 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.981954 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc22e58-ddce-4866-b17d-8672d775fc4a-istiod-ca-cert\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.982230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.982107 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-workload-certs\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.982230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.982128 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-workload-socket\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.982394 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.982374 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-data\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.983921 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.983896 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-envoy\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:50.984440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:50.984416 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-podinfo\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:51.006772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:51.006740 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrnnc\" (UniqueName: \"kubernetes.io/projected/4cc22e58-ddce-4866-b17d-8672d775fc4a-kube-api-access-jrnnc\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:51.007002 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:51.006977 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-token\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:51.111372 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:51.111280 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:51.754675 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:15:51.754639 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc22e58_ddce_4866_b17d_8672d775fc4a.slice/crio-55e2c8a8f6588fa936ff10c780ce3bf92dfd70938f1d81415877fd38145db331 WatchSource:0}: Error finding container 55e2c8a8f6588fa936ff10c780ce3bf92dfd70938f1d81415877fd38145db331: Status 404 returned error can't find the container with id 55e2c8a8f6588fa936ff10c780ce3bf92dfd70938f1d81415877fd38145db331 Oct 29 23:15:51.757506 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:51.757465 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:15:51.757637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:51.757537 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:15:51.757637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:51.757575 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:15:51.757996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:51.757968 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5"] Oct 29 23:15:52.131268 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:52.131125 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" event={"ID":"4cc22e58-ddce-4866-b17d-8672d775fc4a","Type":"ContainerStarted","Data":"a8c12da6975a29ad7545bec1427b4689b37026be94b62c4ac2af49f3f2e02053"} Oct 29 23:15:52.131268 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:52.131242 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" event={"ID":"4cc22e58-ddce-4866-b17d-8672d775fc4a","Type":"ContainerStarted","Data":"55e2c8a8f6588fa936ff10c780ce3bf92dfd70938f1d81415877fd38145db331"} Oct 29 23:15:52.186119 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:52.186060 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" podStartSLOduration=2.186045148 podStartE2EDuration="2.186045148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:52.183675512 +0000 UTC m=+1998.835174045" watchObservedRunningTime="2026-10-29 23:15:52.186045148 +0000 UTC m=+1998.837543677" Oct 29 23:15:53.111646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:53.111606 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:53.113300 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:53.113269 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 29 23:15:53.113417 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:53.113323 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" podUID="4cc22e58-ddce-4866-b17d-8672d775fc4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 29 23:15:54.112003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:54.111965 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 29 23:15:54.112532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:54.112033 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" podUID="4cc22e58-ddce-4866-b17d-8672d775fc4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 29 23:15:55.112431 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:55.112395 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 29 23:15:55.112843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:55.112454 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" podUID="4cc22e58-ddce-4866-b17d-8672d775fc4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 29 23:15:56.118207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:56.114502 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 29 23:15:56.118207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:56.114579 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" podUID="4cc22e58-ddce-4866-b17d-8672d775fc4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 29 23:15:57.112247 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:57.112211 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 29 23:15:57.112432 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:57.112264 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" podUID="4cc22e58-ddce-4866-b17d-8672d775fc4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 29 23:15:58.128212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:58.117610 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 29 23:15:58.128212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:58.117668 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" podUID="4cc22e58-ddce-4866-b17d-8672d775fc4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 29 23:15:59.116446 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:59.116412 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:59.116944 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:59.116922 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:15:59.117586 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:15:59.117568 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:16:02.119707 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.119673 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m"] Oct 29 23:16:02.120726 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.120702 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a53b0b99-b590-4aff-aef5-4793abe1f1d4" containerName="cleanup" Oct 29 23:16:02.120856 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.120727 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53b0b99-b590-4aff-aef5-4793abe1f1d4" containerName="cleanup" Oct 29 23:16:02.121003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.120745 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5f4756a4-cf22-42c0-89b7-c861adeed9ca" containerName="cleanup" Oct 29 23:16:02.121003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.120977 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f4756a4-cf22-42c0-89b7-c861adeed9ca" containerName="cleanup" Oct 29 23:16:02.121131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.121005 5057 cpu_manager.go:414] "containerMap: removing container" podUID="028e4624-bd70-4f79-8026-a998ba9b9f58" containerName="cleanup" Oct 29 23:16:02.121131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.121013 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="028e4624-bd70-4f79-8026-a998ba9b9f58" containerName="cleanup" Oct 29 23:16:02.121131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.121046 5057 cpu_manager.go:414] "containerMap: removing container" podUID="35ad588a-24ca-4823-9d3d-3f281139f7e6" containerName="cleanup" Oct 29 23:16:02.121131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.121053 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35ad588a-24ca-4823-9d3d-3f281139f7e6" containerName="cleanup" Oct 29 23:16:02.121131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.121068 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a89a235c-5f5e-4b9f-b0b4-1bd345e2afae" containerName="cleanup" Oct 29 23:16:02.121131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.121075 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a89a235c-5f5e-4b9f-b0b4-1bd345e2afae" containerName="cleanup" Oct 29 23:16:02.121476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.121328 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53b0b99-b590-4aff-aef5-4793abe1f1d4" containerName="cleanup" Oct 29 23:16:02.121476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.121357 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f4756a4-cf22-42c0-89b7-c861adeed9ca" containerName="cleanup" Oct 29 23:16:02.121476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.121368 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="35ad588a-24ca-4823-9d3d-3f281139f7e6" containerName="cleanup" Oct 29 23:16:02.121476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.121379 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="483a170f-cb23-4c9c-9d95-47a7be2344ff" containerName="cleanup" Oct 29 23:16:02.121476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.121395 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a89a235c-5f5e-4b9f-b0b4-1bd345e2afae" containerName="cleanup" Oct 29 23:16:02.154912 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.154866 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m"] Oct 29 23:16:02.155171 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.154999 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.172054 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.172021 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-2b0108-openshift-default-dockercfg-v8tck\"" Oct 29 23:16:02.229069 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.229026 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.229295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.229133 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.229366 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.229309 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.229530 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.229508 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7cfw\" (UniqueName: \"kubernetes.io/projected/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-kube-api-access-g7cfw\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.229598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.229552 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.229728 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.229703 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.229852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.229834 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.229908 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.229890 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.229953 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.229940 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.348215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.338847 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7cfw\" (UniqueName: \"kubernetes.io/projected/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-kube-api-access-g7cfw\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.348215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.338952 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.348215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.339036 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.348215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.339161 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.348215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.339291 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.348215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.339359 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.348215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.339652 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.348215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.339705 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.348215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.339823 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.348899 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.348656 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.357225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.350123 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.357616 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.357571 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.369210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.358331 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.471209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.449931 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.471209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.450154 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.509285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.480632 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.584226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.583685 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.696227 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.695993 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7cfw\" (UniqueName: \"kubernetes.io/projected/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-kube-api-access-g7cfw\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:02.789262 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:02.778762 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:03.344216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:03.341093 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m"] Oct 29 23:16:03.376218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:03.375488 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:16:03.376218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:03.375623 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:16:03.376218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:03.375699 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:16:03.709754 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:03.709711 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" event={"ID":"f99eb02d-30df-4333-b4a3-ea8b5ff576cf","Type":"ContainerStarted","Data":"c0745d030ae4907be62e5ae3868c41192fc928a3852e69e880a5b6640cda036a"} Oct 29 23:16:03.709963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:03.709782 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" event={"ID":"f99eb02d-30df-4333-b4a3-ea8b5ff576cf","Type":"ContainerStarted","Data":"9aba5d888f25fb191f6e404e7694e6943c9314f0b6d3a3e47d594c776fcbecac"} Oct 29 23:16:03.764414 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:03.764351 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" podStartSLOduration=1.764329601 podStartE2EDuration="1.764329601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:03.758673877 +0000 UTC m=+2010.410172406" watchObservedRunningTime="2026-10-29 23:16:03.764329601 +0000 UTC m=+2010.415828133" Oct 29 23:16:03.779752 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:03.779718 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:04.060088 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:16:04.059999 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:16:04.785833 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:04.785798 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:05.726984 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:05.726531 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:05.727697 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:05.727675 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:16:10.999091 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:10.982046 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7896-ef6a11"] Oct 29 23:16:11.007134 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:11.007103 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-ef6a11" Oct 29 23:16:11.018440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:11.018391 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-ef6a11"] Oct 29 23:16:11.094477 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:11.094432 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xdwj4\"" Oct 29 23:16:11.130155 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:11.130118 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq4t9\" (UniqueName: \"kubernetes.io/projected/d14d97af-0358-4491-8ee2-5c483a224616-kube-api-access-tq4t9\") pod \"test-curl-7896-ef6a11\" (UID: \"d14d97af-0358-4491-8ee2-5c483a224616\") " pod="opendatahub/test-curl-7896-ef6a11" Oct 29 23:16:11.233067 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:11.233021 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq4t9\" (UniqueName: \"kubernetes.io/projected/d14d97af-0358-4491-8ee2-5c483a224616-kube-api-access-tq4t9\") pod \"test-curl-7896-ef6a11\" (UID: \"d14d97af-0358-4491-8ee2-5c483a224616\") " pod="opendatahub/test-curl-7896-ef6a11" Oct 29 23:16:11.258196 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:11.258108 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq4t9\" (UniqueName: \"kubernetes.io/projected/d14d97af-0358-4491-8ee2-5c483a224616-kube-api-access-tq4t9\") pod \"test-curl-7896-ef6a11\" (UID: \"d14d97af-0358-4491-8ee2-5c483a224616\") " pod="opendatahub/test-curl-7896-ef6a11" Oct 29 23:16:11.347882 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:11.347845 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-ef6a11" Oct 29 23:16:11.699647 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:11.699620 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-ef6a11"] Oct 29 23:16:11.700009 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:16:11.699981 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd14d97af_0358_4491_8ee2_5c483a224616.slice/crio-805f78e477457d985873a372a701bc961f9e6a8898b3cf774c596172f7435dc8 WatchSource:0}: Error finding container 805f78e477457d985873a372a701bc961f9e6a8898b3cf774c596172f7435dc8: Status 404 returned error can't find the container with id 805f78e477457d985873a372a701bc961f9e6a8898b3cf774c596172f7435dc8 Oct 29 23:16:11.702044 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:11.702024 5057 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:16:11.878074 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:11.878038 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-ef6a11" event={"ID":"d14d97af-0358-4491-8ee2-5c483a224616","Type":"ContainerStarted","Data":"805f78e477457d985873a372a701bc961f9e6a8898b3cf774c596172f7435dc8"} Oct 29 23:16:14.909160 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:14.909126 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-ef6a11" event={"ID":"d14d97af-0358-4491-8ee2-5c483a224616","Type":"ContainerStarted","Data":"7820881f27e0d302faca4f92d28821d3ee3c550919603d704d351aeea679e7d7"} Oct 29 23:16:14.953881 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:14.953825 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7896-ef6a11" podStartSLOduration=2.439237299 podStartE2EDuration="4.953810632s" totalImagesPullingTime="2.514573333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:14.95056842 +0000 UTC m=+2021.602066933" watchObservedRunningTime="2026-10-29 23:16:14.953810632 +0000 UTC m=+2021.605309162" Oct 29 23:16:15.851820 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:15.851788 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7896-ef6a11"] Oct 29 23:16:15.858712 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:15.858678 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7896-ef6a11"] Oct 29 23:16:16.793516 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:16.793477 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb"] Oct 29 23:16:16.795960 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:16.795929 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d14d97af-0358-4491-8ee2-5c483a224616" containerName="test-curl-7896-ef6a11" Oct 29 23:16:16.796136 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:16.796121 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d14d97af-0358-4491-8ee2-5c483a224616" containerName="test-curl-7896-ef6a11" Oct 29 23:16:16.796774 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:16.796752 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d14d97af-0358-4491-8ee2-5c483a224616" containerName="test-curl-7896-ef6a11" Oct 29 23:16:16.811913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:16.811885 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:16.851561 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:16.849978 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-a4f2db-openshift-default-dockercfg-f5sgq\"" Oct 29 23:16:16.870024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:16.869989 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb"] Oct 29 23:16:16.914407 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:16.914374 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-istio-data\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:16.914601 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:16.914520 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-workload-certs\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:16.914601 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:16.914587 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-workload-socket\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:16.914724 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:16.914695 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4104e017-b65b-404f-8226-71df8ca5c903-istio-token\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:16.914820 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:16.914805 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-istio-envoy\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:16.914868 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:16.914846 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4104e017-b65b-404f-8226-71df8ca5c903-istio-podinfo\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:16.914905 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:16.914872 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl5gz\" (UniqueName: \"kubernetes.io/projected/4104e017-b65b-404f-8226-71df8ca5c903-kube-api-access-kl5gz\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:16.915022 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:16.915002 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-credential-socket\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:16.915080 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:16.915069 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4104e017-b65b-404f-8226-71df8ca5c903-istiod-ca-cert\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:16.928096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:16.928062 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7896-ef6a11" podUID="d14d97af-0358-4491-8ee2-5c483a224616" containerName="test-curl-7896-ef6a11" containerID="cri-o://7820881f27e0d302faca4f92d28821d3ee3c550919603d704d351aeea679e7d7" gracePeriod=2 Oct 29 23:16:16.943663 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:16:16.943619 5057 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7896-ef6a11\" is forbidden: User \"system:node:ip-10-0-135-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-80.ec2.internal' and this object" podUID="d14d97af-0358-4491-8ee2-5c483a224616" pod="opendatahub/test-curl-7896-ef6a11" Oct 29 23:16:17.034845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.028287 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-credential-socket\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:17.034845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.028431 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4104e017-b65b-404f-8226-71df8ca5c903-istiod-ca-cert\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:17.034845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.028588 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-istio-data\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:17.034845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.028730 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-workload-certs\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:17.034845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.028813 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-workload-socket\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:17.034845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.028873 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4104e017-b65b-404f-8226-71df8ca5c903-istio-token\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:17.034845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.028959 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-istio-envoy\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:17.034845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.029021 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4104e017-b65b-404f-8226-71df8ca5c903-istio-podinfo\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:17.034845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.029065 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl5gz\" (UniqueName: \"kubernetes.io/projected/4104e017-b65b-404f-8226-71df8ca5c903-kube-api-access-kl5gz\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:17.034845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.029976 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-credential-socket\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:17.034845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.030657 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4104e017-b65b-404f-8226-71df8ca5c903-istiod-ca-cert\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:17.034845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.030948 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-istio-data\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:17.034845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.031216 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-workload-certs\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:17.034845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.031502 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-workload-socket\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:17.058259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.053800 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-istio-envoy\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:17.091901 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.085468 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl5gz\" (UniqueName: \"kubernetes.io/projected/4104e017-b65b-404f-8226-71df8ca5c903-kube-api-access-kl5gz\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:17.093228 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.093176 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4104e017-b65b-404f-8226-71df8ca5c903-istio-token\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:17.117870 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.110006 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4104e017-b65b-404f-8226-71df8ca5c903-istio-podinfo\") pod \"e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:17.194216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.180563 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:17.807336 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:16:17.807304 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4104e017_b65b_404f_8226_71df8ca5c903.slice/crio-f877e453847c930e52fb84d46364106a8f3ffc07e90aa952f47bc9e3bf5ffb90 WatchSource:0}: Error finding container f877e453847c930e52fb84d46364106a8f3ffc07e90aa952f47bc9e3bf5ffb90: Status 404 returned error can't find the container with id f877e453847c930e52fb84d46364106a8f3ffc07e90aa952f47bc9e3bf5ffb90 Oct 29 23:16:17.808495 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.808469 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb"] Oct 29 23:16:17.810373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.810341 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:16:17.810454 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.810405 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:16:17.810454 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.810435 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:16:17.942083 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.941868 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" event={"ID":"4104e017-b65b-404f-8226-71df8ca5c903","Type":"ContainerStarted","Data":"7f930136d37f6277742099dba83ed718bf209a41345210ea80deac27433f6d9e"} Oct 29 23:16:17.942083 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.942002 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" event={"ID":"4104e017-b65b-404f-8226-71df8ca5c903","Type":"ContainerStarted","Data":"f877e453847c930e52fb84d46364106a8f3ffc07e90aa952f47bc9e3bf5ffb90"} Oct 29 23:16:17.997052 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:17.996987 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" podStartSLOduration=1.996967509 podStartE2EDuration="1.996967509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:17.99090645 +0000 UTC m=+2024.642404980" watchObservedRunningTime="2026-10-29 23:16:17.996967509 +0000 UTC m=+2024.648466043" Oct 29 23:16:18.200213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:18.186180 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:18.200213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:18.186338 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 29 23:16:18.200213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:18.186381 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" podUID="4104e017-b65b-404f-8226-71df8ca5c903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 29 23:16:19.155076 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:19.155045 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-ef6a11" Oct 29 23:16:19.187340 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:19.187257 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:19.326159 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:19.326120 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d14d97af-0358-4491-8ee2-5c483a224616-kube-api-access-tq4t9\" (UniqueName: \"kubernetes.io/projected/d14d97af-0358-4491-8ee2-5c483a224616-kube-api-access-tq4t9\") pod \"d14d97af-0358-4491-8ee2-5c483a224616\" (UID: \"d14d97af-0358-4491-8ee2-5c483a224616\") " Oct 29 23:16:19.328562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:19.328528 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d14d97af-0358-4491-8ee2-5c483a224616-kube-api-access-tq4t9" pod "d14d97af-0358-4491-8ee2-5c483a224616" (UID: "d14d97af-0358-4491-8ee2-5c483a224616"). InnerVolumeSpecName "kube-api-access-tq4t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:19.428200 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:19.428152 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq4t9\" (UniqueName: \"kubernetes.io/projected/d14d97af-0358-4491-8ee2-5c483a224616-kube-api-access-tq4t9\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:16:19.491095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:19.491012 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d14d97af-0358-4491-8ee2-5c483a224616" path="/var/lib/kubelet/pods/d14d97af-0358-4491-8ee2-5c483a224616/volumes" Oct 29 23:16:19.969122 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:19.969083 5057 generic.go:356] "Generic (PLEG): container finished" podID="d14d97af-0358-4491-8ee2-5c483a224616" containerID="7820881f27e0d302faca4f92d28821d3ee3c550919603d704d351aeea679e7d7" exitCode=137 Oct 29 23:16:19.969350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:19.969142 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-ef6a11" Oct 29 23:16:19.969350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:19.969231 5057 scope.go:122] "RemoveContainer" containerID="7820881f27e0d302faca4f92d28821d3ee3c550919603d704d351aeea679e7d7" Oct 29 23:16:19.969580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:19.969562 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:19.970692 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:19.970670 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:16:19.981036 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:19.980425 5057 scope.go:122] "RemoveContainer" containerID="7820881f27e0d302faca4f92d28821d3ee3c550919603d704d351aeea679e7d7" Oct 29 23:16:19.981036 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:16:19.980885 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7820881f27e0d302faca4f92d28821d3ee3c550919603d704d351aeea679e7d7\": container with ID starting with 7820881f27e0d302faca4f92d28821d3ee3c550919603d704d351aeea679e7d7 not found: ID does not exist" containerID="7820881f27e0d302faca4f92d28821d3ee3c550919603d704d351aeea679e7d7" Oct 29 23:16:19.981036 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:19.980934 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7820881f27e0d302faca4f92d28821d3ee3c550919603d704d351aeea679e7d7"} err="failed to get container status \"7820881f27e0d302faca4f92d28821d3ee3c550919603d704d351aeea679e7d7\": rpc error: code = NotFound desc = could not find container \"7820881f27e0d302faca4f92d28821d3ee3c550919603d704d351aeea679e7d7\": container with ID starting with 7820881f27e0d302faca4f92d28821d3ee3c550919603d704d351aeea679e7d7 not found: ID does not exist" Oct 29 23:16:25.673719 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:25.673678 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:16:27.706288 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:27.704615 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg"] Oct 29 23:16:27.758261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:27.753367 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg"] Oct 29 23:16:27.758261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:27.753489 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:27.775692 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:27.774635 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-4ad17e\"/\"auth-test-model-4ad17e-kserve-self-signed-certs\"" Oct 29 23:16:27.791211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:27.789284 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-4ad17e\"/\"kube-root-ca.crt\"" Oct 29 23:16:27.791211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:27.789402 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-4ad17e\"/\"default-dockercfg-nmb5m\"" Oct 29 23:16:27.791211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:27.789506 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-4ad17e\"/\"openshift-service-ca.crt\"" Oct 29 23:16:27.998269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:27.979835 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74ngh\" (UniqueName: \"kubernetes.io/projected/0014c879-9ec4-4197-984a-f10a20b6510e-kube-api-access-74ngh\") pod \"auth-test-model-4ad17e-kserve-779764f6cc-bk4zg\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:28.013436 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:28.013390 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-dshm\") pod \"auth-test-model-4ad17e-kserve-779764f6cc-bk4zg\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:28.021274 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:28.013790 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0014c879-9ec4-4197-984a-f10a20b6510e-tls-certs\") pod \"auth-test-model-4ad17e-kserve-779764f6cc-bk4zg\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:28.115173 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:28.115129 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-model-cache\") pod \"auth-test-model-4ad17e-kserve-779764f6cc-bk4zg\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:28.130645 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:28.130599 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-home\") pod \"auth-test-model-4ad17e-kserve-779764f6cc-bk4zg\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:28.136208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:28.131040 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-tmp-dir\") pod \"auth-test-model-4ad17e-kserve-779764f6cc-bk4zg\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:28.307598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:28.256644 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74ngh\" (UniqueName: \"kubernetes.io/projected/0014c879-9ec4-4197-984a-f10a20b6510e-kube-api-access-74ngh\") pod \"auth-test-model-4ad17e-kserve-779764f6cc-bk4zg\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:28.308016 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:28.307976 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-dshm\") pod \"auth-test-model-4ad17e-kserve-779764f6cc-bk4zg\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:28.308404 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:28.308382 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0014c879-9ec4-4197-984a-f10a20b6510e-tls-certs\") pod \"auth-test-model-4ad17e-kserve-779764f6cc-bk4zg\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:28.308637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:28.308621 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-model-cache\") pod \"auth-test-model-4ad17e-kserve-779764f6cc-bk4zg\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:28.308823 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:28.308804 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-home\") pod \"auth-test-model-4ad17e-kserve-779764f6cc-bk4zg\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:28.309007 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:28.308990 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-tmp-dir\") pod \"auth-test-model-4ad17e-kserve-779764f6cc-bk4zg\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:28.309701 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:28.309680 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-tmp-dir\") pod \"auth-test-model-4ad17e-kserve-779764f6cc-bk4zg\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:28.339255 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:28.330081 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-dshm\") pod \"auth-test-model-4ad17e-kserve-779764f6cc-bk4zg\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:28.344463 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:28.344434 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74ngh\" (UniqueName: \"kubernetes.io/projected/0014c879-9ec4-4197-984a-f10a20b6510e-kube-api-access-74ngh\") pod \"auth-test-model-4ad17e-kserve-779764f6cc-bk4zg\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:28.345599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:28.345270 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-home\") pod \"auth-test-model-4ad17e-kserve-779764f6cc-bk4zg\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:28.348894 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:28.348859 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-model-cache\") pod \"auth-test-model-4ad17e-kserve-779764f6cc-bk4zg\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:28.362579 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:28.362512 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0014c879-9ec4-4197-984a-f10a20b6510e-tls-certs\") pod \"auth-test-model-4ad17e-kserve-779764f6cc-bk4zg\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:28.556475 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:28.556441 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:28.969576 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:16:28.969311 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0014c879_9ec4_4197_984a_f10a20b6510e.slice/crio-1141f5208030b70373e1a771a5d651998ac26e1cfefd1256802fd512cb038f09 WatchSource:0}: Error finding container 1141f5208030b70373e1a771a5d651998ac26e1cfefd1256802fd512cb038f09: Status 404 returned error can't find the container with id 1141f5208030b70373e1a771a5d651998ac26e1cfefd1256802fd512cb038f09 Oct 29 23:16:28.980549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:28.980510 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg"] Oct 29 23:16:29.377999 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:29.377904 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" event={"ID":"0014c879-9ec4-4197-984a-f10a20b6510e","Type":"ContainerStarted","Data":"3f3b44a8f33f9f68d26b313ccbe640da7bbff70904f3d0b54a367e7aac23ab3e"} Oct 29 23:16:29.377999 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:29.377962 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" event={"ID":"0014c879-9ec4-4197-984a-f10a20b6510e","Type":"ContainerStarted","Data":"1141f5208030b70373e1a771a5d651998ac26e1cfefd1256802fd512cb038f09"} Oct 29 23:16:29.460815 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:29.460747 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" podStartSLOduration=2.460729001 podStartE2EDuration="2.460729001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:29.428706275 +0000 UTC m=+2036.080204806" watchObservedRunningTime="2026-10-29 23:16:29.460729001 +0000 UTC m=+2036.112227532" Oct 29 23:16:29.581197 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:29.557740 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:34.015862 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:16:34.015816 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:16:38.036369 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:38.036327 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:16:38.036834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:38.036394 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" podUID="5c287175-5866-4c13-b806-5839582839cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:16:38.557401 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:38.557374 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:38.571050 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:38.571023 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:39.502468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:39.502430 5057 generic.go:356] "Generic (PLEG): container finished" podID="5c287175-5866-4c13-b806-5839582839cc" containerID="081524ad795f0d79617a18403c6e36507788c2a4e9441e0832a1e3ccfbb6fbd0" exitCode=0 Oct 29 23:16:39.503008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:39.502488 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" event={"ID":"5c287175-5866-4c13-b806-5839582839cc","Type":"ContainerDied","Data":"081524ad795f0d79617a18403c6e36507788c2a4e9441e0832a1e3ccfbb6fbd0"} Oct 29 23:16:39.503614 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:39.503436 5057 scope.go:122] "RemoveContainer" containerID="081524ad795f0d79617a18403c6e36507788c2a4e9441e0832a1e3ccfbb6fbd0" Oct 29 23:16:39.505555 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:39.505526 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:16:39.505622 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:39.505604 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:16:39.505672 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:39.505642 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:16:39.517543 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:39.517516 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:16:40.513562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:40.513530 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" event={"ID":"5c287175-5866-4c13-b806-5839582839cc","Type":"ContainerStarted","Data":"fc0b40b5017859e1b0803ac831d27092d19986023b421294bf6e31b19f365e5d"} Oct 29 23:16:40.617438 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:40.617405 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:16:40.617622 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:40.617584 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 29 23:16:40.617732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:40.617644 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" podUID="5c287175-5866-4c13-b806-5839582839cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 29 23:16:41.617161 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:41.617131 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 29 23:16:41.617567 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:41.617180 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" podUID="5c287175-5866-4c13-b806-5839582839cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 29 23:16:42.617415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:42.617379 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 29 23:16:42.617800 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:42.617446 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" podUID="5c287175-5866-4c13-b806-5839582839cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 29 23:16:43.627214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:43.621566 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 29 23:16:43.627214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:43.621644 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" podUID="5c287175-5866-4c13-b806-5839582839cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 29 23:16:44.620106 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:44.618701 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 29 23:16:44.620106 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:44.618786 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" podUID="5c287175-5866-4c13-b806-5839582839cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 29 23:16:45.616936 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:45.616898 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 29 23:16:45.617385 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:45.616951 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" podUID="5c287175-5866-4c13-b806-5839582839cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 29 23:16:46.617236 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:46.617199 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 29 23:16:46.617640 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:46.617267 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" podUID="5c287175-5866-4c13-b806-5839582839cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 29 23:16:47.715211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:47.709553 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:16:47.715211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:47.710808 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:16:47.725215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:16:47.716654 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:17:03.914762 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:17:03.914721 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:17:25.463065 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.463015 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl"] Oct 29 23:17:25.463643 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.463282 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" podUID="b5d56fdf-4c0d-4243-980d-ae3e958544c2" containerName="istio-proxy" containerID="cri-o://ee07dcf24644b423262a3c8f4d06aae1c9fc6cb4bfd329333e8ddc8fa57e7ae6" gracePeriod=30 Oct 29 23:17:25.490622 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.490577 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr"] Oct 29 23:17:25.490822 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.490796 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" podUID="bff30ab0-9c2d-41f0-902f-685ae8c67c0a" containerName="istio-proxy" containerID="cri-o://2c4a58c98e1ddb110e7aae38ac189b7bdf9be0b36b0d4f6bc8e46138fcf8220c" gracePeriod=30 Oct 29 23:17:25.638119 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.638083 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4"] Oct 29 23:17:25.644775 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.644750 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.663651 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.663312 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-dea9cb-openshift-default-dockercfg-dgcrj\"" Oct 29 23:17:25.677442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.677385 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4"] Oct 29 23:17:25.688416 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.688378 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-workload-socket\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.688573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.688441 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-token\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.688620 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.688560 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istiod-ca-cert\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.688665 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.688641 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-credential-socket\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.688727 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.688709 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-podinfo\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.688987 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.688970 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-workload-certs\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.689026 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.689011 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-envoy\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.689084 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.689070 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjkjv\" (UniqueName: \"kubernetes.io/projected/fd6e00d3-c546-4fec-82fd-98c420fcca2d-kube-api-access-qjkjv\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.689138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.689124 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-data\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.798400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.797892 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-workload-certs\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.798400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.797947 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-envoy\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.798400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.798006 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjkjv\" (UniqueName: \"kubernetes.io/projected/fd6e00d3-c546-4fec-82fd-98c420fcca2d-kube-api-access-qjkjv\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.798400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.798052 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-data\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.798400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.798163 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-workload-socket\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.798400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.798239 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-token\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.798400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.798272 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istiod-ca-cert\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.798400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.798335 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-credential-socket\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.798400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.798398 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-podinfo\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.799639 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.799609 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-data\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.823376 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.806491 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istiod-ca-cert\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.824350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.824286 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-workload-certs\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.824772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.824720 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-credential-socket\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.825843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.825731 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-workload-socket\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.982214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.970469 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-podinfo\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.987094 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.986377 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjkjv\" (UniqueName: \"kubernetes.io/projected/fd6e00d3-c546-4fec-82fd-98c420fcca2d-kube-api-access-qjkjv\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.987094 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.986649 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-envoy\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:25.987094 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:25.987036 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-token\") pod \"e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:26.292227 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:26.287510 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:26.731527 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:17:26.731495 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6e00d3_c546_4fec_82fd_98c420fcca2d.slice/crio-59e3e10d5c75e2bb7b3b69fa6193cbf3165afe97486839eb48c08c0163e7c7e6 WatchSource:0}: Error finding container 59e3e10d5c75e2bb7b3b69fa6193cbf3165afe97486839eb48c08c0163e7c7e6: Status 404 returned error can't find the container with id 59e3e10d5c75e2bb7b3b69fa6193cbf3165afe97486839eb48c08c0163e7c7e6 Oct 29 23:17:26.734231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:26.734169 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4"] Oct 29 23:17:26.735648 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:26.735604 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:17:26.735748 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:26.735696 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:17:26.735748 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:26.735742 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:17:27.286642 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:27.286587 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" event={"ID":"fd6e00d3-c546-4fec-82fd-98c420fcca2d","Type":"ContainerStarted","Data":"59396594e1193249bbbe8a320ddba6f2d133c40082212650848d4e4589440d1d"} Oct 29 23:17:27.286833 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:27.286670 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" event={"ID":"fd6e00d3-c546-4fec-82fd-98c420fcca2d","Type":"ContainerStarted","Data":"59e3e10d5c75e2bb7b3b69fa6193cbf3165afe97486839eb48c08c0163e7c7e6"} Oct 29 23:17:27.292972 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:27.292949 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:27.294711 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:27.294676 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 29 23:17:27.294827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:27.294725 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" podUID="fd6e00d3-c546-4fec-82fd-98c420fcca2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 29 23:17:27.344634 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:27.344578 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" podStartSLOduration=2.344563351 podStartE2EDuration="2.344563351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:27.339218627 +0000 UTC m=+2093.990717158" watchObservedRunningTime="2026-10-29 23:17:27.344563351 +0000 UTC m=+2093.996061939" Oct 29 23:17:28.297996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:28.297968 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:29.303318 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:29.303261 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:29.304406 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:29.304376 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:17:30.936011 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:30.935971 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:17:31.023516 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.023421 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:17:31.095378 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.095343 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d56fdf-4c0d-4243-980d-ae3e958544c2-kube-api-access-626k9\" (UniqueName: \"kubernetes.io/projected/b5d56fdf-4c0d-4243-980d-ae3e958544c2-kube-api-access-626k9\") pod \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " Oct 29 23:17:31.095598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.095513 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-workload-certs\") pod \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " Oct 29 23:17:31.095598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.095557 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-credential-socket\") pod \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " Oct 29 23:17:31.095598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.095595 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-workload-socket\") pod \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " Oct 29 23:17:31.095787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.095631 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istiod-ca-cert\") pod \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " Oct 29 23:17:31.095787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.095715 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-podinfo\") pod \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " Oct 29 23:17:31.095787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.095749 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-envoy\") pod \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " Oct 29 23:17:31.095787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.095782 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-workload-certs\") pod \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " Oct 29 23:17:31.096021 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.095824 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-kube-api-access-bx8ww\" (UniqueName: \"kubernetes.io/projected/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-kube-api-access-bx8ww\") pod \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " Oct 29 23:17:31.096021 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.095899 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-workload-socket\") pod \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " Oct 29 23:17:31.096021 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.095933 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-credential-socket" pod "bff30ab0-9c2d-41f0-902f-685ae8c67c0a" (UID: "bff30ab0-9c2d-41f0-902f-685ae8c67c0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:31.096021 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.095937 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-token\" (UniqueName: \"kubernetes.io/projected/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-token\") pod \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " Oct 29 23:17:31.096338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.096048 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-envoy\") pod \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " Oct 29 23:17:31.096338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.096077 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-credential-socket\") pod \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " Oct 29 23:17:31.096338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.096174 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-data\") pod \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " Oct 29 23:17:31.096338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.096262 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-podinfo\") pod \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\" (UID: \"b5d56fdf-4c0d-4243-980d-ae3e958544c2\") " Oct 29 23:17:31.096338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.096309 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-token\" (UniqueName: \"kubernetes.io/projected/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-token\") pod \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " Oct 29 23:17:31.096338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.096330 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istiod-ca-cert\") pod \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " Oct 29 23:17:31.098405 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.097472 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:17:31.098405 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.097744 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-credential-socket" pod "b5d56fdf-4c0d-4243-980d-ae3e958544c2" (UID: "b5d56fdf-4c0d-4243-980d-ae3e958544c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:31.098405 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.097818 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istiod-ca-cert" pod "bff30ab0-9c2d-41f0-902f-685ae8c67c0a" (UID: "bff30ab0-9c2d-41f0-902f-685ae8c67c0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:17:31.098405 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.098087 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-workload-certs" pod "b5d56fdf-4c0d-4243-980d-ae3e958544c2" (UID: "b5d56fdf-4c0d-4243-980d-ae3e958544c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:31.098812 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.098640 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-token" pod "b5d56fdf-4c0d-4243-980d-ae3e958544c2" (UID: "b5d56fdf-4c0d-4243-980d-ae3e958544c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:17:31.098812 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.098806 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-workload-certs" pod "bff30ab0-9c2d-41f0-902f-685ae8c67c0a" (UID: "bff30ab0-9c2d-41f0-902f-685ae8c67c0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:31.099336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.099311 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istiod-ca-cert" pod "b5d56fdf-4c0d-4243-980d-ae3e958544c2" (UID: "b5d56fdf-4c0d-4243-980d-ae3e958544c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:17:31.099700 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.099681 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-workload-socket" pod "bff30ab0-9c2d-41f0-902f-685ae8c67c0a" (UID: "bff30ab0-9c2d-41f0-902f-685ae8c67c0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:31.100001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.099984 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-workload-socket" pod "b5d56fdf-4c0d-4243-980d-ae3e958544c2" (UID: "b5d56fdf-4c0d-4243-980d-ae3e958544c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:31.100349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.100328 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-data" pod "b5d56fdf-4c0d-4243-980d-ae3e958544c2" (UID: "b5d56fdf-4c0d-4243-980d-ae3e958544c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:31.101874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.101730 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-envoy" pod "b5d56fdf-4c0d-4243-980d-ae3e958544c2" (UID: "b5d56fdf-4c0d-4243-980d-ae3e958544c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:31.102502 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.102476 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-kube-api-access-bx8ww" pod "bff30ab0-9c2d-41f0-902f-685ae8c67c0a" (UID: "bff30ab0-9c2d-41f0-902f-685ae8c67c0a"). InnerVolumeSpecName "kube-api-access-bx8ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:17:31.102927 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.102639 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d56fdf-4c0d-4243-980d-ae3e958544c2-kube-api-access-626k9" pod "b5d56fdf-4c0d-4243-980d-ae3e958544c2" (UID: "b5d56fdf-4c0d-4243-980d-ae3e958544c2"). InnerVolumeSpecName "kube-api-access-626k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:17:31.102927 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.102723 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-podinfo" pod "bff30ab0-9c2d-41f0-902f-685ae8c67c0a" (UID: "bff30ab0-9c2d-41f0-902f-685ae8c67c0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:17:31.102927 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.102824 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-envoy" pod "bff30ab0-9c2d-41f0-902f-685ae8c67c0a" (UID: "bff30ab0-9c2d-41f0-902f-685ae8c67c0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:31.103767 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.103743 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-podinfo" pod "b5d56fdf-4c0d-4243-980d-ae3e958544c2" (UID: "b5d56fdf-4c0d-4243-980d-ae3e958544c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:17:31.103894 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.103878 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-token" pod "bff30ab0-9c2d-41f0-902f-685ae8c67c0a" (UID: "bff30ab0-9c2d-41f0-902f-685ae8c67c0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:17:31.199897 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.198902 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-data\") pod \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\" (UID: \"bff30ab0-9c2d-41f0-902f-685ae8c67c0a\") " Oct 29 23:17:31.200096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.199949 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:17:31.200096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.199975 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:17:31.200096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.199988 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:17:31.200096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.199999 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:17:31.200096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.200012 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-626k9\" (UniqueName: \"kubernetes.io/projected/b5d56fdf-4c0d-4243-980d-ae3e958544c2-kube-api-access-626k9\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:17:31.200096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.200025 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:17:31.200096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.200039 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:17:31.200096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.200054 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:17:31.200096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.200067 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:17:31.200096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.200081 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:17:31.200096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.200094 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:17:31.200562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.200088 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-data" pod "bff30ab0-9c2d-41f0-902f-685ae8c67c0a" (UID: "bff30ab0-9c2d-41f0-902f-685ae8c67c0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:31.200562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.200109 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx8ww\" (UniqueName: \"kubernetes.io/projected/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-kube-api-access-bx8ww\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:17:31.200562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.200122 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:17:31.200562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.200136 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:17:31.200562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.200150 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:17:31.200562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.200162 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d56fdf-4c0d-4243-980d-ae3e958544c2-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:17:31.301790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.301709 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bff30ab0-9c2d-41f0-902f-685ae8c67c0a-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:17:31.320841 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.320807 5057 generic.go:356] "Generic (PLEG): container finished" podID="b5d56fdf-4c0d-4243-980d-ae3e958544c2" containerID="ee07dcf24644b423262a3c8f4d06aae1c9fc6cb4bfd329333e8ddc8fa57e7ae6" exitCode=0 Oct 29 23:17:31.321013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.320860 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" Oct 29 23:17:31.321013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.320886 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" event={"ID":"b5d56fdf-4c0d-4243-980d-ae3e958544c2","Type":"ContainerDied","Data":"ee07dcf24644b423262a3c8f4d06aae1c9fc6cb4bfd329333e8ddc8fa57e7ae6"} Oct 29 23:17:31.321013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.320942 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl" event={"ID":"b5d56fdf-4c0d-4243-980d-ae3e958544c2","Type":"ContainerDied","Data":"58265bed9b73c4c7026c0be4ab96512afc4ed2ba62372e1d93de597f71d2e18a"} Oct 29 23:17:31.321013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.320970 5057 scope.go:122] "RemoveContainer" containerID="ee07dcf24644b423262a3c8f4d06aae1c9fc6cb4bfd329333e8ddc8fa57e7ae6" Oct 29 23:17:31.322739 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.322714 5057 generic.go:356] "Generic (PLEG): container finished" podID="bff30ab0-9c2d-41f0-902f-685ae8c67c0a" containerID="2c4a58c98e1ddb110e7aae38ac189b7bdf9be0b36b0d4f6bc8e46138fcf8220c" exitCode=0 Oct 29 23:17:31.322914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.322769 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" Oct 29 23:17:31.322914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.322780 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" event={"ID":"bff30ab0-9c2d-41f0-902f-685ae8c67c0a","Type":"ContainerDied","Data":"2c4a58c98e1ddb110e7aae38ac189b7bdf9be0b36b0d4f6bc8e46138fcf8220c"} Oct 29 23:17:31.322914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.322823 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr" event={"ID":"bff30ab0-9c2d-41f0-902f-685ae8c67c0a","Type":"ContainerDied","Data":"02b07883a15980d1c259fce9983aa3c7c51e146ccaf460f6187dd3743a408231"} Oct 29 23:17:31.334425 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.334402 5057 scope.go:122] "RemoveContainer" containerID="ee07dcf24644b423262a3c8f4d06aae1c9fc6cb4bfd329333e8ddc8fa57e7ae6" Oct 29 23:17:31.334794 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:17:31.334759 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee07dcf24644b423262a3c8f4d06aae1c9fc6cb4bfd329333e8ddc8fa57e7ae6\": container with ID starting with ee07dcf24644b423262a3c8f4d06aae1c9fc6cb4bfd329333e8ddc8fa57e7ae6 not found: ID does not exist" containerID="ee07dcf24644b423262a3c8f4d06aae1c9fc6cb4bfd329333e8ddc8fa57e7ae6" Oct 29 23:17:31.334889 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.334802 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee07dcf24644b423262a3c8f4d06aae1c9fc6cb4bfd329333e8ddc8fa57e7ae6"} err="failed to get container status \"ee07dcf24644b423262a3c8f4d06aae1c9fc6cb4bfd329333e8ddc8fa57e7ae6\": rpc error: code = NotFound desc = could not find container \"ee07dcf24644b423262a3c8f4d06aae1c9fc6cb4bfd329333e8ddc8fa57e7ae6\": container with ID starting with ee07dcf24644b423262a3c8f4d06aae1c9fc6cb4bfd329333e8ddc8fa57e7ae6 not found: ID does not exist" Oct 29 23:17:31.334889 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.334826 5057 scope.go:122] "RemoveContainer" containerID="2c4a58c98e1ddb110e7aae38ac189b7bdf9be0b36b0d4f6bc8e46138fcf8220c" Oct 29 23:17:31.345545 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.345525 5057 scope.go:122] "RemoveContainer" containerID="2c4a58c98e1ddb110e7aae38ac189b7bdf9be0b36b0d4f6bc8e46138fcf8220c" Oct 29 23:17:31.345854 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:17:31.345824 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c4a58c98e1ddb110e7aae38ac189b7bdf9be0b36b0d4f6bc8e46138fcf8220c\": container with ID starting with 2c4a58c98e1ddb110e7aae38ac189b7bdf9be0b36b0d4f6bc8e46138fcf8220c not found: ID does not exist" containerID="2c4a58c98e1ddb110e7aae38ac189b7bdf9be0b36b0d4f6bc8e46138fcf8220c" Oct 29 23:17:31.345962 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.345862 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c4a58c98e1ddb110e7aae38ac189b7bdf9be0b36b0d4f6bc8e46138fcf8220c"} err="failed to get container status \"2c4a58c98e1ddb110e7aae38ac189b7bdf9be0b36b0d4f6bc8e46138fcf8220c\": rpc error: code = NotFound desc = could not find container \"2c4a58c98e1ddb110e7aae38ac189b7bdf9be0b36b0d4f6bc8e46138fcf8220c\": container with ID starting with 2c4a58c98e1ddb110e7aae38ac189b7bdf9be0b36b0d4f6bc8e46138fcf8220c not found: ID does not exist" Oct 29 23:17:31.381615 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.381584 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr"] Oct 29 23:17:31.395013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.394986 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kkcfr"] Oct 29 23:17:31.422792 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.422766 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl"] Oct 29 23:17:31.427293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.427269 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-xp2tl"] Oct 29 23:17:31.487330 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.487293 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d56fdf-4c0d-4243-980d-ae3e958544c2" path="/var/lib/kubelet/pods/b5d56fdf-4c0d-4243-980d-ae3e958544c2/volumes" Oct 29 23:17:31.488028 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.487990 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bff30ab0-9c2d-41f0-902f-685ae8c67c0a" path="/var/lib/kubelet/pods/bff30ab0-9c2d-41f0-902f-685ae8c67c0a/volumes" Oct 29 23:17:31.737701 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.737664 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs"] Oct 29 23:17:31.738537 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.738520 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b5d56fdf-4c0d-4243-980d-ae3e958544c2" containerName="istio-proxy" Oct 29 23:17:31.738600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.738538 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d56fdf-4c0d-4243-980d-ae3e958544c2" containerName="istio-proxy" Oct 29 23:17:31.738600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.738586 5057 cpu_manager.go:414] "containerMap: removing container" podUID="bff30ab0-9c2d-41f0-902f-685ae8c67c0a" containerName="istio-proxy" Oct 29 23:17:31.738600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.738591 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff30ab0-9c2d-41f0-902f-685ae8c67c0a" containerName="istio-proxy" Oct 29 23:17:31.738759 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.738740 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d56fdf-4c0d-4243-980d-ae3e958544c2" containerName="istio-proxy" Oct 29 23:17:31.738759 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.738763 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff30ab0-9c2d-41f0-902f-685ae8c67c0a" containerName="istio-proxy" Oct 29 23:17:31.744450 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.744430 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs" Oct 29 23:17:31.764446 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.764423 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-dea9cb-dockercfg-xpjqs\"" Oct 29 23:17:31.764762 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.764739 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-dea9cb\"" Oct 29 23:17:31.769753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.769728 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs"] Oct 29 23:17:31.835210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.834441 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eaa234f-64b4-4698-bf92-c0c03de0e701-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs\" (UID: \"1eaa234f-64b4-4698-bf92-c0c03de0e701\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs" Oct 29 23:17:31.835210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.834887 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xjj6\" (UniqueName: \"kubernetes.io/projected/1eaa234f-64b4-4698-bf92-c0c03de0e701-kube-api-access-7xjj6\") pod \"payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs\" (UID: \"1eaa234f-64b4-4698-bf92-c0c03de0e701\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs" Oct 29 23:17:31.964209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.943870 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xjj6\" (UniqueName: \"kubernetes.io/projected/1eaa234f-64b4-4698-bf92-c0c03de0e701-kube-api-access-7xjj6\") pod \"payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs\" (UID: \"1eaa234f-64b4-4698-bf92-c0c03de0e701\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs" Oct 29 23:17:31.964209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.944013 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eaa234f-64b4-4698-bf92-c0c03de0e701-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs\" (UID: \"1eaa234f-64b4-4698-bf92-c0c03de0e701\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs" Oct 29 23:17:31.964209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:31.944965 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eaa234f-64b4-4698-bf92-c0c03de0e701-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs\" (UID: \"1eaa234f-64b4-4698-bf92-c0c03de0e701\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs" Oct 29 23:17:32.172349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:32.155891 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xjj6\" (UniqueName: \"kubernetes.io/projected/1eaa234f-64b4-4698-bf92-c0c03de0e701-kube-api-access-7xjj6\") pod \"payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs\" (UID: \"1eaa234f-64b4-4698-bf92-c0c03de0e701\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs" Oct 29 23:17:32.360164 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:32.360139 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs" Oct 29 23:17:32.362958 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:32.362925 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf"] Oct 29 23:17:32.369139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:32.369116 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf" Oct 29 23:17:32.387103 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:32.387079 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf"] Oct 29 23:17:32.492742 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:32.492677 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0787860-0815-4f18-9424-cd3ab9944810-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf\" (UID: \"d0787860-0815-4f18-9424-cd3ab9944810\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf" Oct 29 23:17:32.493051 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:32.493027 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx6k9\" (UniqueName: \"kubernetes.io/projected/d0787860-0815-4f18-9424-cd3ab9944810-kube-api-access-gx6k9\") pod \"payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf\" (UID: \"d0787860-0815-4f18-9424-cd3ab9944810\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf" Oct 29 23:17:32.595290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:32.595246 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0787860-0815-4f18-9424-cd3ab9944810-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf\" (UID: \"d0787860-0815-4f18-9424-cd3ab9944810\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf" Oct 29 23:17:32.595496 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:32.595376 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx6k9\" (UniqueName: \"kubernetes.io/projected/d0787860-0815-4f18-9424-cd3ab9944810-kube-api-access-gx6k9\") pod \"payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf\" (UID: \"d0787860-0815-4f18-9424-cd3ab9944810\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf" Oct 29 23:17:32.596541 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:32.596507 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0787860-0815-4f18-9424-cd3ab9944810-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf\" (UID: \"d0787860-0815-4f18-9424-cd3ab9944810\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf" Oct 29 23:17:32.621486 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:32.621460 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx6k9\" (UniqueName: \"kubernetes.io/projected/d0787860-0815-4f18-9424-cd3ab9944810-kube-api-access-gx6k9\") pod \"payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf\" (UID: \"d0787860-0815-4f18-9424-cd3ab9944810\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf" Oct 29 23:17:32.683029 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:17:32.682997 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eaa234f_64b4_4698_bf92_c0c03de0e701.slice/crio-c0e2bc85c438b771a98bbc6ea06d1150be60735ce421f7352f610a644c6a935a WatchSource:0}: Error finding container c0e2bc85c438b771a98bbc6ea06d1150be60735ce421f7352f610a644c6a935a: Status 404 returned error can't find the container with id c0e2bc85c438b771a98bbc6ea06d1150be60735ce421f7352f610a644c6a935a Oct 29 23:17:32.683997 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:32.683968 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs"] Oct 29 23:17:32.731244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:32.731210 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf" Oct 29 23:17:33.048494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:33.048466 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf"] Oct 29 23:17:33.382635 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:33.382481 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs" event={"ID":"1eaa234f-64b4-4698-bf92-c0c03de0e701","Type":"ContainerStarted","Data":"75e0fcab00d9a748a17ec1fd29df30a2973fba70513d57ac4f91a2c3daa502e3"} Oct 29 23:17:33.382892 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:33.382863 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs" event={"ID":"1eaa234f-64b4-4698-bf92-c0c03de0e701","Type":"ContainerStarted","Data":"c0e2bc85c438b771a98bbc6ea06d1150be60735ce421f7352f610a644c6a935a"} Oct 29 23:17:33.382994 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:33.382915 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs" Oct 29 23:17:33.385622 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:33.385594 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf" event={"ID":"d0787860-0815-4f18-9424-cd3ab9944810","Type":"ContainerStarted","Data":"dec44c7059e947903408bdb9c4742e11b9028d53e21517fc0d6d4200ac218879"} Oct 29 23:17:33.385766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:33.385642 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf" event={"ID":"d0787860-0815-4f18-9424-cd3ab9944810","Type":"ContainerStarted","Data":"5aa4cd2d7cf4703d900300f0bc0a4fede5fa358bd9a9d6d62a692b6708988332"} Oct 29 23:17:33.385766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:33.385671 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf" Oct 29 23:17:33.430201 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:33.430147 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs" podStartSLOduration=2.430133512 podStartE2EDuration="2.430133512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:33.426031689 +0000 UTC m=+2100.077530220" watchObservedRunningTime="2026-10-29 23:17:33.430133512 +0000 UTC m=+2100.081632041" Oct 29 23:17:33.470524 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:33.470471 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf" podStartSLOduration=2.470456464 podStartE2EDuration="2.470456464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:33.469460053 +0000 UTC m=+2100.120958586" watchObservedRunningTime="2026-10-29 23:17:33.470456464 +0000 UTC m=+2100.121954994" Oct 29 23:17:33.918254 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:17:33.918219 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:17:34.899921 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:34.899891 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:17:34.932346 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:34.932318 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:17:42.141207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:42.140879 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:17:43.051175 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.051141 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49"] Oct 29 23:17:43.064143 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.064113 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.087376 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.087344 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49"] Oct 29 23:17:43.093287 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.093258 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-383f0e-openshift-default-dockercfg-4kdgt\"" Oct 29 23:17:43.282872 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.282837 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-istio-data\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.283579 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.283538 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-credential-socket\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.283796 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.283779 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-workload-certs\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.354469 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.354377 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-workload-socket\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.354776 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.354748 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-istio-envoy\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.355148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.355123 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f255ae1a-010b-4f50-9414-5b09b507c085-istio-podinfo\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.477249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.469844 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f255ae1a-010b-4f50-9414-5b09b507c085-istiod-ca-cert\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.477249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.469954 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-credential-socket\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.477249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.470013 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2jm4\" (UniqueName: \"kubernetes.io/projected/f255ae1a-010b-4f50-9414-5b09b507c085-kube-api-access-r2jm4\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.477249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.470058 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-workload-certs\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.477249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.470283 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-workload-socket\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.477249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.470322 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-istio-envoy\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.477249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.470490 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f255ae1a-010b-4f50-9414-5b09b507c085-istio-token\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.477249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.470557 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f255ae1a-010b-4f50-9414-5b09b507c085-istio-podinfo\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.477249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.470662 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-istio-data\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.482475 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.482422 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-istio-data\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.482921 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.482888 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-credential-socket\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.483412 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.483272 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-workload-certs\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.483591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.483382 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-workload-socket\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.544601 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.544013 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-istio-envoy\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.598015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.591852 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f255ae1a-010b-4f50-9414-5b09b507c085-istio-token\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.598015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.592117 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f255ae1a-010b-4f50-9414-5b09b507c085-istiod-ca-cert\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.599079 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.599050 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f255ae1a-010b-4f50-9414-5b09b507c085-istiod-ca-cert\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.599293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.599271 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2jm4\" (UniqueName: \"kubernetes.io/projected/f255ae1a-010b-4f50-9414-5b09b507c085-kube-api-access-r2jm4\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.634263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.606967 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f255ae1a-010b-4f50-9414-5b09b507c085-istio-podinfo\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.683861 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.683824 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2jm4\" (UniqueName: \"kubernetes.io/projected/f255ae1a-010b-4f50-9414-5b09b507c085-kube-api-access-r2jm4\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.685715 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.685686 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f255ae1a-010b-4f50-9414-5b09b507c085-istio-token\") pod \"e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:43.703974 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:43.694687 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:44.597213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:44.544413 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf" Oct 29 23:17:44.694209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:44.687371 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs" Oct 29 23:17:45.096785 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:45.036944 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:17:45.096785 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:45.037426 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:17:45.238238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:45.225896 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded" Oct 29 23:17:45.309239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:45.290858 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:17:45.703689 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:45.703644 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49"] Oct 29 23:17:45.704674 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:45.704629 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:17:45.705135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:45.705108 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:17:45.705347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:45.705324 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:17:46.472446 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:46.472409 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" event={"ID":"f255ae1a-010b-4f50-9414-5b09b507c085","Type":"ContainerStarted","Data":"12f9dddeba1de36d011970ce9fc27ed207ddb84b6c30fd3e37d49eafd3881b41"} Oct 29 23:17:46.472664 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:46.472476 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" event={"ID":"f255ae1a-010b-4f50-9414-5b09b507c085","Type":"ContainerStarted","Data":"f572f2131e825f22ae0caf37701d14e615791bce802682ff5db0a92aa72d41e3"} Oct 29 23:17:46.562077 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:46.562023 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" podStartSLOduration=3.562006951 podStartE2EDuration="3.562006951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:46.543663652 +0000 UTC m=+2113.195162184" watchObservedRunningTime="2026-10-29 23:17:46.562006951 +0000 UTC m=+2113.213505481" Oct 29 23:17:46.701306 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:46.701272 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:46.707385 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:46.707352 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:47.484946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:47.484913 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:47.485125 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:47.484982 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:17:56.744833 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:56.744794 5057 generic.go:356] "Generic (PLEG): container finished" podID="4104e017-b65b-404f-8226-71df8ca5c903" containerID="7f930136d37f6277742099dba83ed718bf209a41345210ea80deac27433f6d9e" exitCode=0 Oct 29 23:17:56.745711 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:56.745587 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" event={"ID":"4104e017-b65b-404f-8226-71df8ca5c903","Type":"ContainerDied","Data":"7f930136d37f6277742099dba83ed718bf209a41345210ea80deac27433f6d9e"} Oct 29 23:17:56.746514 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:56.746490 5057 scope.go:122] "RemoveContainer" containerID="7f930136d37f6277742099dba83ed718bf209a41345210ea80deac27433f6d9e" Oct 29 23:17:56.750164 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:56.750127 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:17:56.750905 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:56.750878 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:17:56.751079 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:56.751060 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:17:57.759437 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:57.759395 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" event={"ID":"4104e017-b65b-404f-8226-71df8ca5c903","Type":"ContainerStarted","Data":"59eba49db777c8dfcd6a586e44e7b15743786122ad608008d103438840d62135"} Oct 29 23:17:58.182384 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:58.182348 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:17:58.183755 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:58.183727 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 29 23:17:58.183885 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:58.183777 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" podUID="4104e017-b65b-404f-8226-71df8ca5c903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 29 23:17:59.182661 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:59.182616 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 29 23:17:59.183096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:17:59.182681 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" podUID="4104e017-b65b-404f-8226-71df8ca5c903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 29 23:18:00.181845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:00.181800 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 29 23:18:00.182072 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:00.181867 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" podUID="4104e017-b65b-404f-8226-71df8ca5c903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 29 23:18:00.355357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:00.355268 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm"] Oct 29 23:18:00.470727 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:00.463668 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k"] Oct 29 23:18:01.206208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:01.205434 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 29 23:18:01.206208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:01.205498 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" podUID="4104e017-b65b-404f-8226-71df8ca5c903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 29 23:18:02.182451 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:02.182411 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 29 23:18:02.182966 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:02.182477 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" podUID="4104e017-b65b-404f-8226-71df8ca5c903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 29 23:18:03.182115 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:03.182083 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 29 23:18:03.182311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:03.182138 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" podUID="4104e017-b65b-404f-8226-71df8ca5c903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 29 23:18:03.918719 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:18:03.918679 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:18:04.189978 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:04.189941 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:18:04.191214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:04.190444 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:18:04.191657 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:04.191630 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:18:05.357659 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:05.357617 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm" podUID="4528935e-c7f6-4c37-a477-2e006cd53fb3" containerName="payload-processing" containerID="cri-o://96a680cf042ae9437491fe29d3946401d6d47304da7e9546ce366833d20b08e9" gracePeriod=25 Oct 29 23:18:05.472074 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:05.472022 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k" podUID="3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6" containerName="payload-pre-processing" containerID="cri-o://3dbb0b88993c1220c4f7eb32a33308c1636dd25a330ba94cb5efaafc1e239c67" gracePeriod=25 Oct 29 23:18:12.326027 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:12.325991 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9"] Oct 29 23:18:12.326557 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:12.326230 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" podUID="a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" containerName="istio-proxy" containerID="cri-o://c7309b2929fd6e102fa50ad5d34ff54ecfafc11b388ed5440de25dabf06e8e72" gracePeriod=30 Oct 29 23:18:12.342868 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:12.342811 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r"] Oct 29 23:18:12.343109 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:12.343078 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" podUID="71a86092-f8c0-4608-8c15-9112358560f4" containerName="istio-proxy" containerID="cri-o://6d0abf1a5fbea40ad6d1d6cc419d9843cff6e04c56cba2c5e6184221f1498aca" gracePeriod=30 Oct 29 23:18:12.350148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:12.350115 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm"] Oct 29 23:18:12.350413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:12.350369 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" podUID="b68d1f22-af34-4928-9ea2-c89e5a881ff0" containerName="istio-proxy" containerID="cri-o://f538417a45d71fa19caf6747690d8161965a422cb4c18d8b2201843426a76ef4" gracePeriod=30 Oct 29 23:18:12.945466 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:12.945429 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k" podUID="3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:18:12.945664 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:12.945436 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm" podUID="4528935e-c7f6-4c37-a477-2e006cd53fb3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:18:17.702233 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.702194 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:18:17.784760 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.783975 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:18:17.790371 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.790349 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:18:17.874585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.874491 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-workload-certs\") pod \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " Oct 29 23:18:17.874585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.874558 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71a86092-f8c0-4608-8c15-9112358560f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a86092-f8c0-4608-8c15-9112358560f4-istiod-ca-cert\") pod \"71a86092-f8c0-4608-8c15-9112358560f4\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " Oct 29 23:18:17.874843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.874669 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-istio-envoy\") pod \"71a86092-f8c0-4608-8c15-9112358560f4\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " Oct 29 23:18:17.874843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.874732 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-envoy\") pod \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " Oct 29 23:18:17.874843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.874807 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-workload-socket\") pod \"71a86092-f8c0-4608-8c15-9112358560f4\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " Oct 29 23:18:17.875020 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.874901 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istiod-ca-cert\") pod \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " Oct 29 23:18:17.875020 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.874952 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-workload-certs\") pod \"71a86092-f8c0-4608-8c15-9112358560f4\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " Oct 29 23:18:17.875132 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.875102 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-workload-certs\") pod \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " Oct 29 23:18:17.875239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.875137 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a86092-f8c0-4608-8c15-9112358560f4-kube-api-access-x8h5k\" (UniqueName: \"kubernetes.io/projected/71a86092-f8c0-4608-8c15-9112358560f4-kube-api-access-x8h5k\") pod \"71a86092-f8c0-4608-8c15-9112358560f4\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " Oct 29 23:18:17.875312 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.875255 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-workload-socket\") pod \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " Oct 29 23:18:17.875312 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.875304 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a86092-f8c0-4608-8c15-9112358560f4-istio-token\" (UniqueName: \"kubernetes.io/projected/71a86092-f8c0-4608-8c15-9112358560f4-istio-token\") pod \"71a86092-f8c0-4608-8c15-9112358560f4\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " Oct 29 23:18:17.875430 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.875338 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-podinfo\") pod \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " Oct 29 23:18:17.875430 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.875383 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68d1f22-af34-4928-9ea2-c89e5a881ff0-kube-api-access-fhrgg\" (UniqueName: \"kubernetes.io/projected/b68d1f22-af34-4928-9ea2-c89e5a881ff0-kube-api-access-fhrgg\") pod \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " Oct 29 23:18:17.875537 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.875454 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-data\") pod \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " Oct 29 23:18:17.875537 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.875486 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-credential-socket\") pod \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " Oct 29 23:18:17.875653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.875568 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-workload-certs" pod "a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" (UID: "a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.875653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.875574 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-credential-socket\") pod \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " Oct 29 23:18:17.875766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.875668 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istiod-ca-cert\") pod \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " Oct 29 23:18:17.875766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.875721 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-data\") pod \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " Oct 29 23:18:17.875875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.875780 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71a86092-f8c0-4608-8c15-9112358560f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a86092-f8c0-4608-8c15-9112358560f4-istio-podinfo\") pod \"71a86092-f8c0-4608-8c15-9112358560f4\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " Oct 29 23:18:17.875938 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.875873 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-credential-socket" pod "b68d1f22-af34-4928-9ea2-c89e5a881ff0" (UID: "b68d1f22-af34-4928-9ea2-c89e5a881ff0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.875938 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.875883 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-podinfo\") pod \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " Oct 29 23:18:17.876057 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.875950 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-token\" (UniqueName: \"kubernetes.io/projected/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-token\") pod \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " Oct 29 23:18:17.876057 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.875986 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-kube-api-access-67wkh\" (UniqueName: \"kubernetes.io/projected/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-kube-api-access-67wkh\") pod \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " Oct 29 23:18:17.876057 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.876039 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-workload-socket\") pod \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " Oct 29 23:18:17.876246 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.876076 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-credential-socket\") pod \"71a86092-f8c0-4608-8c15-9112358560f4\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " Oct 29 23:18:17.876246 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.876133 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-token\" (UniqueName: \"kubernetes.io/projected/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-token\") pod \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\" (UID: \"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4\") " Oct 29 23:18:17.876362 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.876244 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-istio-data\") pod \"71a86092-f8c0-4608-8c15-9112358560f4\" (UID: \"71a86092-f8c0-4608-8c15-9112358560f4\") " Oct 29 23:18:17.876362 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.876327 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-envoy\") pod \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\" (UID: \"b68d1f22-af34-4928-9ea2-c89e5a881ff0\") " Oct 29 23:18:17.877563 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.877468 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.877563 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.877495 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.879016 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.878983 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istiod-ca-cert" pod "b68d1f22-af34-4928-9ea2-c89e5a881ff0" (UID: "b68d1f22-af34-4928-9ea2-c89e5a881ff0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:17.879373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.879344 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-data" pod "a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" (UID: "a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.879821 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.879653 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-workload-socket" pod "a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" (UID: "a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.879821 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.879697 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-envoy" pod "a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" (UID: "a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.880040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.879883 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-workload-socket" pod "71a86092-f8c0-4608-8c15-9112358560f4" (UID: "71a86092-f8c0-4608-8c15-9112358560f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.880237 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.880212 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istiod-ca-cert" pod "a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" (UID: "a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:17.880475 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.880452 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-workload-certs" pod "71a86092-f8c0-4608-8c15-9112358560f4" (UID: "71a86092-f8c0-4608-8c15-9112358560f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.881084 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.880665 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-workload-certs" pod "b68d1f22-af34-4928-9ea2-c89e5a881ff0" (UID: "b68d1f22-af34-4928-9ea2-c89e5a881ff0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.881084 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.881040 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71a86092-f8c0-4608-8c15-9112358560f4-istio-podinfo" pod "71a86092-f8c0-4608-8c15-9112358560f4" (UID: "71a86092-f8c0-4608-8c15-9112358560f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:17.881326 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.881175 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-credential-socket" pod "71a86092-f8c0-4608-8c15-9112358560f4" (UID: "71a86092-f8c0-4608-8c15-9112358560f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.881394 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.881338 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-podinfo" pod "b68d1f22-af34-4928-9ea2-c89e5a881ff0" (UID: "b68d1f22-af34-4928-9ea2-c89e5a881ff0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:17.881460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.881435 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-token" pod "b68d1f22-af34-4928-9ea2-c89e5a881ff0" (UID: "b68d1f22-af34-4928-9ea2-c89e5a881ff0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:17.881567 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.881541 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-istio-data" pod "71a86092-f8c0-4608-8c15-9112358560f4" (UID: "71a86092-f8c0-4608-8c15-9112358560f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.881866 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.881819 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a86092-f8c0-4608-8c15-9112358560f4-istiod-ca-cert" pod "71a86092-f8c0-4608-8c15-9112358560f4" (UID: "71a86092-f8c0-4608-8c15-9112358560f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:17.882798 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.882158 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-istio-envoy" pod "71a86092-f8c0-4608-8c15-9112358560f4" (UID: "71a86092-f8c0-4608-8c15-9112358560f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.882798 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.882222 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-data" pod "b68d1f22-af34-4928-9ea2-c89e5a881ff0" (UID: "b68d1f22-af34-4928-9ea2-c89e5a881ff0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.882798 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.882499 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-workload-socket" pod "b68d1f22-af34-4928-9ea2-c89e5a881ff0" (UID: "b68d1f22-af34-4928-9ea2-c89e5a881ff0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.882798 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.882570 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-envoy" pod "b68d1f22-af34-4928-9ea2-c89e5a881ff0" (UID: "b68d1f22-af34-4928-9ea2-c89e5a881ff0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.882798 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.882775 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-credential-socket" pod "a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" (UID: "a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.883477 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.883451 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-token" pod "a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" (UID: "a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:17.883589 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.883492 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a86092-f8c0-4608-8c15-9112358560f4-istio-token" pod "71a86092-f8c0-4608-8c15-9112358560f4" (UID: "71a86092-f8c0-4608-8c15-9112358560f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:17.883589 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.883552 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-kube-api-access-67wkh" pod "a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" (UID: "a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4"). InnerVolumeSpecName "kube-api-access-67wkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:17.884468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.884442 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a86092-f8c0-4608-8c15-9112358560f4-kube-api-access-x8h5k" pod "71a86092-f8c0-4608-8c15-9112358560f4" (UID: "71a86092-f8c0-4608-8c15-9112358560f4"). InnerVolumeSpecName "kube-api-access-x8h5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:17.884720 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.884699 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-podinfo" pod "a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" (UID: "a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:17.885320 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.885298 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68d1f22-af34-4928-9ea2-c89e5a881ff0-kube-api-access-fhrgg" pod "b68d1f22-af34-4928-9ea2-c89e5a881ff0" (UID: "b68d1f22-af34-4928-9ea2-c89e5a881ff0"). InnerVolumeSpecName "kube-api-access-fhrgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:17.979219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979165 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979218 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979234 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979246 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979259 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8h5k\" (UniqueName: \"kubernetes.io/projected/71a86092-f8c0-4608-8c15-9112358560f4-kube-api-access-x8h5k\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979273 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979286 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a86092-f8c0-4608-8c15-9112358560f4-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979299 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979313 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhrgg\" (UniqueName: \"kubernetes.io/projected/b68d1f22-af34-4928-9ea2-c89e5a881ff0-kube-api-access-fhrgg\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979327 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979339 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979353 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979367 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979378 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a86092-f8c0-4608-8c15-9112358560f4-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979391 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979404 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979416 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67wkh\" (UniqueName: \"kubernetes.io/projected/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-kube-api-access-67wkh\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979429 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979443 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979456 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979468 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979481 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68d1f22-af34-4928-9ea2-c89e5a881ff0-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979492 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a86092-f8c0-4608-8c15-9112358560f4-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979503 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a86092-f8c0-4608-8c15-9112358560f4-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.979922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.979515 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.984409 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.984380 5057 generic.go:356] "Generic (PLEG): container finished" podID="b68d1f22-af34-4928-9ea2-c89e5a881ff0" containerID="f538417a45d71fa19caf6747690d8161965a422cb4c18d8b2201843426a76ef4" exitCode=0 Oct 29 23:18:17.984538 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.984423 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" Oct 29 23:18:17.984538 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.984464 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" event={"ID":"b68d1f22-af34-4928-9ea2-c89e5a881ff0","Type":"ContainerDied","Data":"f538417a45d71fa19caf6747690d8161965a422cb4c18d8b2201843426a76ef4"} Oct 29 23:18:17.984538 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.984518 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm" event={"ID":"b68d1f22-af34-4928-9ea2-c89e5a881ff0","Type":"ContainerDied","Data":"c2e9c7a6dea80d783b806af4b6929f5222845f3cf47b2781b03c3cf6e9078d74"} Oct 29 23:18:17.984667 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.984547 5057 scope.go:122] "RemoveContainer" containerID="f538417a45d71fa19caf6747690d8161965a422cb4c18d8b2201843426a76ef4" Oct 29 23:18:17.986225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.986084 5057 generic.go:356] "Generic (PLEG): container finished" podID="71a86092-f8c0-4608-8c15-9112358560f4" containerID="6d0abf1a5fbea40ad6d1d6cc419d9843cff6e04c56cba2c5e6184221f1498aca" exitCode=0 Oct 29 23:18:17.986225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.986133 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" Oct 29 23:18:17.986225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.986161 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" event={"ID":"71a86092-f8c0-4608-8c15-9112358560f4","Type":"ContainerDied","Data":"6d0abf1a5fbea40ad6d1d6cc419d9843cff6e04c56cba2c5e6184221f1498aca"} Oct 29 23:18:17.986381 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.986226 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r" event={"ID":"71a86092-f8c0-4608-8c15-9112358560f4","Type":"ContainerDied","Data":"57e380d92330fb6733e491d00a6a0f95f3bc53d764625d90a178540ec247f8ec"} Oct 29 23:18:17.991227 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.988114 5057 generic.go:356] "Generic (PLEG): container finished" podID="a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" containerID="c7309b2929fd6e102fa50ad5d34ff54ecfafc11b388ed5440de25dabf06e8e72" exitCode=0 Oct 29 23:18:17.991227 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.988162 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" event={"ID":"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4","Type":"ContainerDied","Data":"c7309b2929fd6e102fa50ad5d34ff54ecfafc11b388ed5440de25dabf06e8e72"} Oct 29 23:18:17.991227 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.988171 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" Oct 29 23:18:17.991227 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:17.988263 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9" event={"ID":"a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4","Type":"ContainerDied","Data":"7f2864fcb92c8ae887de76cb3928e5fce0f3956fd3aa883bc05c67d86ce13a04"} Oct 29 23:18:18.011328 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:18.011296 5057 scope.go:122] "RemoveContainer" containerID="f538417a45d71fa19caf6747690d8161965a422cb4c18d8b2201843426a76ef4" Oct 29 23:18:18.012040 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:18:18.011877 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f538417a45d71fa19caf6747690d8161965a422cb4c18d8b2201843426a76ef4\": container with ID starting with f538417a45d71fa19caf6747690d8161965a422cb4c18d8b2201843426a76ef4 not found: ID does not exist" containerID="f538417a45d71fa19caf6747690d8161965a422cb4c18d8b2201843426a76ef4" Oct 29 23:18:18.012040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:18.011930 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f538417a45d71fa19caf6747690d8161965a422cb4c18d8b2201843426a76ef4"} err="failed to get container status \"f538417a45d71fa19caf6747690d8161965a422cb4c18d8b2201843426a76ef4\": rpc error: code = NotFound desc = could not find container \"f538417a45d71fa19caf6747690d8161965a422cb4c18d8b2201843426a76ef4\": container with ID starting with f538417a45d71fa19caf6747690d8161965a422cb4c18d8b2201843426a76ef4 not found: ID does not exist" Oct 29 23:18:18.012040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:18.011949 5057 scope.go:122] "RemoveContainer" containerID="6d0abf1a5fbea40ad6d1d6cc419d9843cff6e04c56cba2c5e6184221f1498aca" Oct 29 23:18:18.028841 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:18.028815 5057 scope.go:122] "RemoveContainer" containerID="6d0abf1a5fbea40ad6d1d6cc419d9843cff6e04c56cba2c5e6184221f1498aca" Oct 29 23:18:18.029234 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:18:18.029201 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0abf1a5fbea40ad6d1d6cc419d9843cff6e04c56cba2c5e6184221f1498aca\": container with ID starting with 6d0abf1a5fbea40ad6d1d6cc419d9843cff6e04c56cba2c5e6184221f1498aca not found: ID does not exist" containerID="6d0abf1a5fbea40ad6d1d6cc419d9843cff6e04c56cba2c5e6184221f1498aca" Oct 29 23:18:18.029382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:18.029245 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0abf1a5fbea40ad6d1d6cc419d9843cff6e04c56cba2c5e6184221f1498aca"} err="failed to get container status \"6d0abf1a5fbea40ad6d1d6cc419d9843cff6e04c56cba2c5e6184221f1498aca\": rpc error: code = NotFound desc = could not find container \"6d0abf1a5fbea40ad6d1d6cc419d9843cff6e04c56cba2c5e6184221f1498aca\": container with ID starting with 6d0abf1a5fbea40ad6d1d6cc419d9843cff6e04c56cba2c5e6184221f1498aca not found: ID does not exist" Oct 29 23:18:18.029382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:18.029282 5057 scope.go:122] "RemoveContainer" containerID="c7309b2929fd6e102fa50ad5d34ff54ecfafc11b388ed5440de25dabf06e8e72" Oct 29 23:18:18.044637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:18.044619 5057 scope.go:122] "RemoveContainer" containerID="c7309b2929fd6e102fa50ad5d34ff54ecfafc11b388ed5440de25dabf06e8e72" Oct 29 23:18:18.044946 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:18:18.044925 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7309b2929fd6e102fa50ad5d34ff54ecfafc11b388ed5440de25dabf06e8e72\": container with ID starting with c7309b2929fd6e102fa50ad5d34ff54ecfafc11b388ed5440de25dabf06e8e72 not found: ID does not exist" containerID="c7309b2929fd6e102fa50ad5d34ff54ecfafc11b388ed5440de25dabf06e8e72" Oct 29 23:18:18.045005 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:18.044953 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7309b2929fd6e102fa50ad5d34ff54ecfafc11b388ed5440de25dabf06e8e72"} err="failed to get container status \"c7309b2929fd6e102fa50ad5d34ff54ecfafc11b388ed5440de25dabf06e8e72\": rpc error: code = NotFound desc = could not find container \"c7309b2929fd6e102fa50ad5d34ff54ecfafc11b388ed5440de25dabf06e8e72\": container with ID starting with c7309b2929fd6e102fa50ad5d34ff54ecfafc11b388ed5440de25dabf06e8e72 not found: ID does not exist" Oct 29 23:18:18.085260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:18.085227 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm"] Oct 29 23:18:18.092628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:18.092599 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-gqbwm"] Oct 29 23:18:18.131377 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:18.131286 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r"] Oct 29 23:18:18.136698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:18.136666 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-kp68r"] Oct 29 23:18:18.169563 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:18.169531 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9"] Oct 29 23:18:18.173969 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:18.173945 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v46f9"] Oct 29 23:18:19.493660 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:19.493624 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a86092-f8c0-4608-8c15-9112358560f4" path="/var/lib/kubelet/pods/71a86092-f8c0-4608-8c15-9112358560f4/volumes" Oct 29 23:18:19.494286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:19.494268 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" path="/var/lib/kubelet/pods/a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4/volumes" Oct 29 23:18:19.494895 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:19.494878 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b68d1f22-af34-4928-9ea2-c89e5a881ff0" path="/var/lib/kubelet/pods/b68d1f22-af34-4928-9ea2-c89e5a881ff0/volumes" Oct 29 23:18:22.945948 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:22.945585 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k" podUID="3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:18:22.946416 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:22.946333 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm" podUID="4528935e-c7f6-4c37-a477-2e006cd53fb3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:18:25.827014 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:25.821636 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm" Oct 29 23:18:25.854467 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:25.854420 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4528935e-c7f6-4c37-a477-2e006cd53fb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4528935e-c7f6-4c37-a477-2e006cd53fb3-plugins-config-volume\") pod \"4528935e-c7f6-4c37-a477-2e006cd53fb3\" (UID: \"4528935e-c7f6-4c37-a477-2e006cd53fb3\") " Oct 29 23:18:25.855229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:25.855082 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4528935e-c7f6-4c37-a477-2e006cd53fb3-plugins-config-volume" pod "4528935e-c7f6-4c37-a477-2e006cd53fb3" (UID: "4528935e-c7f6-4c37-a477-2e006cd53fb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:25.855650 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:25.855622 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4528935e-c7f6-4c37-a477-2e006cd53fb3-kube-api-access-8bgjb\" (UniqueName: \"kubernetes.io/projected/4528935e-c7f6-4c37-a477-2e006cd53fb3-kube-api-access-8bgjb\") pod \"4528935e-c7f6-4c37-a477-2e006cd53fb3\" (UID: \"4528935e-c7f6-4c37-a477-2e006cd53fb3\") " Oct 29 23:18:25.856662 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:25.856634 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4528935e-c7f6-4c37-a477-2e006cd53fb3-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:25.890252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:25.890214 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4528935e-c7f6-4c37-a477-2e006cd53fb3-kube-api-access-8bgjb" pod "4528935e-c7f6-4c37-a477-2e006cd53fb3" (UID: "4528935e-c7f6-4c37-a477-2e006cd53fb3"). InnerVolumeSpecName "kube-api-access-8bgjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:25.942889 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:25.937724 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k" Oct 29 23:18:25.958177 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:25.958103 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6-kube-api-access-zxkc8\" (UniqueName: \"kubernetes.io/projected/3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6-kube-api-access-zxkc8\") pod \"3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6\" (UID: \"3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6\") " Oct 29 23:18:25.958177 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:25.958169 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6-plugins-config-volume\") pod \"3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6\" (UID: \"3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6\") " Oct 29 23:18:25.958595 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:25.958563 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6-plugins-config-volume" pod "3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6" (UID: "3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:25.958941 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:25.958923 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:25.959011 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:25.958946 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bgjb\" (UniqueName: \"kubernetes.io/projected/4528935e-c7f6-4c37-a477-2e006cd53fb3-kube-api-access-8bgjb\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:25.960371 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:25.960344 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6-kube-api-access-zxkc8" pod "3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6" (UID: "3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6"). InnerVolumeSpecName "kube-api-access-zxkc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:26.062618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:26.061390 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxkc8\" (UniqueName: \"kubernetes.io/projected/3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6-kube-api-access-zxkc8\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:26.071086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:26.071052 5057 generic.go:356] "Generic (PLEG): container finished" podID="4528935e-c7f6-4c37-a477-2e006cd53fb3" containerID="96a680cf042ae9437491fe29d3946401d6d47304da7e9546ce366833d20b08e9" exitCode=0 Oct 29 23:18:26.071294 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:26.071098 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm" Oct 29 23:18:26.071294 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:26.071137 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm" event={"ID":"4528935e-c7f6-4c37-a477-2e006cd53fb3","Type":"ContainerDied","Data":"96a680cf042ae9437491fe29d3946401d6d47304da7e9546ce366833d20b08e9"} Oct 29 23:18:26.071294 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:26.071214 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm" event={"ID":"4528935e-c7f6-4c37-a477-2e006cd53fb3","Type":"ContainerDied","Data":"14a57b58f09af3f0cc8798c96f8bc65d1b4795744c867b88815fe999ed8c937e"} Oct 29 23:18:26.071294 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:26.071245 5057 scope.go:122] "RemoveContainer" containerID="96a680cf042ae9437491fe29d3946401d6d47304da7e9546ce366833d20b08e9" Oct 29 23:18:26.075232 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:26.073586 5057 generic.go:356] "Generic (PLEG): container finished" podID="3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6" containerID="3dbb0b88993c1220c4f7eb32a33308c1636dd25a330ba94cb5efaafc1e239c67" exitCode=0 Oct 29 23:18:26.075232 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:26.073646 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k" Oct 29 23:18:26.075232 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:26.073688 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k" event={"ID":"3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6","Type":"ContainerDied","Data":"3dbb0b88993c1220c4f7eb32a33308c1636dd25a330ba94cb5efaafc1e239c67"} Oct 29 23:18:26.075232 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:26.073740 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k" event={"ID":"3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6","Type":"ContainerDied","Data":"274d2536e3b43ea2d4aa4d591c9199db557274e4a3dc84c516fd4bf22979d155"} Oct 29 23:18:26.085520 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:26.085504 5057 scope.go:122] "RemoveContainer" containerID="96a680cf042ae9437491fe29d3946401d6d47304da7e9546ce366833d20b08e9" Oct 29 23:18:26.085810 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:18:26.085793 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96a680cf042ae9437491fe29d3946401d6d47304da7e9546ce366833d20b08e9\": container with ID starting with 96a680cf042ae9437491fe29d3946401d6d47304da7e9546ce366833d20b08e9 not found: ID does not exist" containerID="96a680cf042ae9437491fe29d3946401d6d47304da7e9546ce366833d20b08e9" Oct 29 23:18:26.085884 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:26.085816 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96a680cf042ae9437491fe29d3946401d6d47304da7e9546ce366833d20b08e9"} err="failed to get container status \"96a680cf042ae9437491fe29d3946401d6d47304da7e9546ce366833d20b08e9\": rpc error: code = NotFound desc = could not find container \"96a680cf042ae9437491fe29d3946401d6d47304da7e9546ce366833d20b08e9\": container with ID starting with 96a680cf042ae9437491fe29d3946401d6d47304da7e9546ce366833d20b08e9 not found: ID does not exist" Oct 29 23:18:26.085884 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:26.085833 5057 scope.go:122] "RemoveContainer" containerID="3dbb0b88993c1220c4f7eb32a33308c1636dd25a330ba94cb5efaafc1e239c67" Oct 29 23:18:26.096103 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:26.096087 5057 scope.go:122] "RemoveContainer" containerID="3dbb0b88993c1220c4f7eb32a33308c1636dd25a330ba94cb5efaafc1e239c67" Oct 29 23:18:26.096409 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:18:26.096391 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dbb0b88993c1220c4f7eb32a33308c1636dd25a330ba94cb5efaafc1e239c67\": container with ID starting with 3dbb0b88993c1220c4f7eb32a33308c1636dd25a330ba94cb5efaafc1e239c67 not found: ID does not exist" containerID="3dbb0b88993c1220c4f7eb32a33308c1636dd25a330ba94cb5efaafc1e239c67" Oct 29 23:18:26.096470 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:26.096415 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dbb0b88993c1220c4f7eb32a33308c1636dd25a330ba94cb5efaafc1e239c67"} err="failed to get container status \"3dbb0b88993c1220c4f7eb32a33308c1636dd25a330ba94cb5efaafc1e239c67\": rpc error: code = NotFound desc = could not find container \"3dbb0b88993c1220c4f7eb32a33308c1636dd25a330ba94cb5efaafc1e239c67\": container with ID starting with 3dbb0b88993c1220c4f7eb32a33308c1636dd25a330ba94cb5efaafc1e239c67 not found: ID does not exist" Oct 29 23:18:26.130443 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:26.130417 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm"] Oct 29 23:18:26.136822 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:26.136799 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-96cd49-747f7df574-sk4jm"] Oct 29 23:18:26.173635 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:26.173599 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k"] Oct 29 23:18:26.177600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:26.177579 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-96cd49-5648b5c7f5-jp92k"] Oct 29 23:18:27.483625 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:27.483593 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6" path="/var/lib/kubelet/pods/3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6/volumes" Oct 29 23:18:27.484082 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:27.484068 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4528935e-c7f6-4c37-a477-2e006cd53fb3" path="/var/lib/kubelet/pods/4528935e-c7f6-4c37-a477-2e006cd53fb3/volumes" Oct 29 23:18:28.847843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.847806 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j"] Oct 29 23:18:28.848922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.848895 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4528935e-c7f6-4c37-a477-2e006cd53fb3" containerName="payload-processing" Oct 29 23:18:28.848922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.848921 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4528935e-c7f6-4c37-a477-2e006cd53fb3" containerName="payload-processing" Oct 29 23:18:28.849139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.848951 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" containerName="istio-proxy" Oct 29 23:18:28.849139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.848958 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" containerName="istio-proxy" Oct 29 23:18:28.849139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.848998 5057 cpu_manager.go:414] "containerMap: removing container" podUID="71a86092-f8c0-4608-8c15-9112358560f4" containerName="istio-proxy" Oct 29 23:18:28.849139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.849007 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a86092-f8c0-4608-8c15-9112358560f4" containerName="istio-proxy" Oct 29 23:18:28.849139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.849055 5057 cpu_manager.go:414] "containerMap: removing container" podUID="3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6" containerName="payload-pre-processing" Oct 29 23:18:28.849139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.849063 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6" containerName="payload-pre-processing" Oct 29 23:18:28.849139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.849086 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b68d1f22-af34-4928-9ea2-c89e5a881ff0" containerName="istio-proxy" Oct 29 23:18:28.849139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.849095 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b68d1f22-af34-4928-9ea2-c89e5a881ff0" containerName="istio-proxy" Oct 29 23:18:28.849564 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.849321 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a69bc5b4-6dc4-4ecf-8acb-f1504ac988c4" containerName="istio-proxy" Oct 29 23:18:28.849564 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.849346 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a86092-f8c0-4608-8c15-9112358560f4" containerName="istio-proxy" Oct 29 23:18:28.849564 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.849358 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae9bc9b-b7ad-4b9b-9d9f-d2edae6d09c6" containerName="payload-pre-processing" Oct 29 23:18:28.849564 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.849373 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b68d1f22-af34-4928-9ea2-c89e5a881ff0" containerName="istio-proxy" Oct 29 23:18:28.849564 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.849391 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4528935e-c7f6-4c37-a477-2e006cd53fb3" containerName="payload-processing" Oct 29 23:18:28.854760 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.854738 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" Oct 29 23:18:28.877039 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.877010 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-383f0e-serving-cert\"" Oct 29 23:18:28.878871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.878840 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j"] Oct 29 23:18:28.914480 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.914445 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j\" (UID: \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" Oct 29 23:18:28.914688 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.914660 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsp9r\" (UniqueName: \"kubernetes.io/projected/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-kube-api-access-hsp9r\") pod \"maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j\" (UID: \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" Oct 29 23:18:28.914822 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.914803 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j\" (UID: \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" Oct 29 23:18:28.914920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:28.914905 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j\" (UID: \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" Oct 29 23:18:29.040911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:29.033635 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j\" (UID: \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" Oct 29 23:18:29.040911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:29.033811 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsp9r\" (UniqueName: \"kubernetes.io/projected/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-kube-api-access-hsp9r\") pod \"maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j\" (UID: \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" Oct 29 23:18:29.040911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:29.033938 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j\" (UID: \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" Oct 29 23:18:29.040911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:29.034087 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j\" (UID: \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" Oct 29 23:18:29.045219 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:18:29.041971 5057 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-serving-cert: secret "maas-api-e2e-shared-b-w5-383f0e-serving-cert" not found Oct 29 23:18:29.045219 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:18:29.042065 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-maas-api-tls podName:9b1937f5-1a2b-4a0d-a42a-636c5b6aafca nodeName:}" failed. No retries permitted until 2026-10-29 23:18:29.542040512 +0000 UTC m=+2156.193539027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-maas-api-tls") pod "maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" (UID: "9b1937f5-1a2b-4a0d-a42a-636c5b6aafca") : secret "maas-api-e2e-shared-b-w5-383f0e-serving-cert" not found Oct 29 23:18:29.091955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:29.088483 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j\" (UID: \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" Oct 29 23:18:29.117339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:29.108748 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j\" (UID: \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" Oct 29 23:18:29.142231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:29.140852 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsp9r\" (UniqueName: \"kubernetes.io/projected/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-kube-api-access-hsp9r\") pod \"maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j\" (UID: \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" Oct 29 23:18:29.554101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:29.554060 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j\" (UID: \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" Oct 29 23:18:29.558021 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:29.557992 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j\" (UID: \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" Oct 29 23:18:29.788412 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:29.781989 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" Oct 29 23:18:30.303376 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:30.297050 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j"] Oct 29 23:18:31.325991 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:31.325948 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" event={"ID":"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca","Type":"ContainerStarted","Data":"e4a1326be837693741933bf2487aa87e45fb25fc07cf0c111420b8f515cc4d6f"} Oct 29 23:18:31.326531 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:31.326020 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" Oct 29 23:18:31.326531 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:31.326047 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" event={"ID":"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca","Type":"ContainerStarted","Data":"121ac122f544f363ec63554f80acb55a271cd77c001babbde7803831397d5229"} Oct 29 23:18:31.375825 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:31.375774 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" podStartSLOduration=3.029008393 podStartE2EDuration="3.375760958s" totalImagesPullingTime="346.752565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:31.374486604 +0000 UTC m=+2158.025985146" watchObservedRunningTime="2026-10-29 23:18:31.375760958 +0000 UTC m=+2158.027259488" Oct 29 23:18:33.914058 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:18:33.914019 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:18:37.344070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:37.344040 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" Oct 29 23:18:37.950695 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:37.950660 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j"] Oct 29 23:18:37.950900 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:37.950878 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" podUID="afeb2ed0-d3ab-4607-91c1-f97eb0662229" containerName="istio-proxy" containerID="cri-o://39be18009970a45af27112fb42775b844e41afabb2858c74ca5f3bff170c2931" gracePeriod=30 Oct 29 23:18:38.544713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:38.544676 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw"] Oct 29 23:18:38.545299 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:38.544926 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" podUID="2ab845c5-deac-48c4-8a7c-b2dbada5d762" containerName="istio-proxy" containerID="cri-o://a73459a04103a5b698341b633722ca5fa92966f8dc1f720304188a58e5c08e6e" gracePeriod=30 Oct 29 23:18:38.559802 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:38.559771 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp"] Oct 29 23:18:38.560029 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:38.560000 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" podUID="59a46795-a139-40f9-9eea-0a4f93fb244f" containerName="istio-proxy" containerID="cri-o://0650927cf63cfd6bd31ad94bfd71b41e23b3059dbe6b629e6d2618fe6259c932" gracePeriod=30 Oct 29 23:18:42.149061 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:42.149017 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-88j5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:18:42.149520 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:42.149087 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" podUID="afeb2ed0-d3ab-4607-91c1-f97eb0662229" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:18:43.261071 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.261044 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 23:18:43.386892 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.386852 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-workload-socket\") pod \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " Oct 29 23:18:43.387135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.387115 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-token\" (UniqueName: \"kubernetes.io/projected/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-token\") pod \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " Oct 29 23:18:43.387232 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.387133 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-workload-socket" pod "afeb2ed0-d3ab-4607-91c1-f97eb0662229" (UID: "afeb2ed0-d3ab-4607-91c1-f97eb0662229"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:43.387310 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.387228 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-envoy\") pod \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " Oct 29 23:18:43.387310 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.387262 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afeb2ed0-d3ab-4607-91c1-f97eb0662229-kube-api-access-nq4dc\" (UniqueName: \"kubernetes.io/projected/afeb2ed0-d3ab-4607-91c1-f97eb0662229-kube-api-access-nq4dc\") pod \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " Oct 29 23:18:43.387423 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.387322 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-credential-socket\") pod \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " Oct 29 23:18:43.387484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.387457 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-data\") pod \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " Oct 29 23:18:43.387611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.387488 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-podinfo\") pod \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " Oct 29 23:18:43.387611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.387537 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-workload-certs\") pod \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " Oct 29 23:18:43.387611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.387570 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istiod-ca-cert\") pod \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\" (UID: \"afeb2ed0-d3ab-4607-91c1-f97eb0662229\") " Oct 29 23:18:43.388076 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.388049 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-credential-socket" pod "afeb2ed0-d3ab-4607-91c1-f97eb0662229" (UID: "afeb2ed0-d3ab-4607-91c1-f97eb0662229"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:43.388345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.388318 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-workload-certs" pod "afeb2ed0-d3ab-4607-91c1-f97eb0662229" (UID: "afeb2ed0-d3ab-4607-91c1-f97eb0662229"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:43.388444 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.388396 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-data" pod "afeb2ed0-d3ab-4607-91c1-f97eb0662229" (UID: "afeb2ed0-d3ab-4607-91c1-f97eb0662229"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:43.388444 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.388416 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istiod-ca-cert" pod "afeb2ed0-d3ab-4607-91c1-f97eb0662229" (UID: "afeb2ed0-d3ab-4607-91c1-f97eb0662229"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:43.388922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.388900 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:43.389023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.388928 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:43.389023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.388944 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:43.389023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.388959 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:43.389023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.388973 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:43.391407 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.391375 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-podinfo" pod "afeb2ed0-d3ab-4607-91c1-f97eb0662229" (UID: "afeb2ed0-d3ab-4607-91c1-f97eb0662229"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:43.391508 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.391455 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afeb2ed0-d3ab-4607-91c1-f97eb0662229-kube-api-access-nq4dc" pod "afeb2ed0-d3ab-4607-91c1-f97eb0662229" (UID: "afeb2ed0-d3ab-4607-91c1-f97eb0662229"). InnerVolumeSpecName "kube-api-access-nq4dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:43.391508 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.391480 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-token" pod "afeb2ed0-d3ab-4607-91c1-f97eb0662229" (UID: "afeb2ed0-d3ab-4607-91c1-f97eb0662229"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:43.391658 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.391636 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-envoy" pod "afeb2ed0-d3ab-4607-91c1-f97eb0662229" (UID: "afeb2ed0-d3ab-4607-91c1-f97eb0662229"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:43.492005 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.491911 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:43.492005 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.491953 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:43.492005 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.491967 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq4dc\" (UniqueName: \"kubernetes.io/projected/afeb2ed0-d3ab-4607-91c1-f97eb0662229-kube-api-access-nq4dc\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:43.492005 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.491989 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afeb2ed0-d3ab-4607-91c1-f97eb0662229-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:43.595767 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.595731 5057 generic.go:356] "Generic (PLEG): container finished" podID="afeb2ed0-d3ab-4607-91c1-f97eb0662229" containerID="39be18009970a45af27112fb42775b844e41afabb2858c74ca5f3bff170c2931" exitCode=0 Oct 29 23:18:43.595920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.595788 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" Oct 29 23:18:43.595920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.595806 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" event={"ID":"afeb2ed0-d3ab-4607-91c1-f97eb0662229","Type":"ContainerDied","Data":"39be18009970a45af27112fb42775b844e41afabb2858c74ca5f3bff170c2931"} Oct 29 23:18:43.595920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.595859 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j" event={"ID":"afeb2ed0-d3ab-4607-91c1-f97eb0662229","Type":"ContainerDied","Data":"4cd04462a694343e7386f9ac3bcff08641ad4fa12eb402938f3941b56b1adcdc"} Oct 29 23:18:43.595920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.595883 5057 scope.go:122] "RemoveContainer" containerID="39be18009970a45af27112fb42775b844e41afabb2858c74ca5f3bff170c2931" Oct 29 23:18:43.676926 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.676486 5057 scope.go:122] "RemoveContainer" containerID="39be18009970a45af27112fb42775b844e41afabb2858c74ca5f3bff170c2931" Oct 29 23:18:43.677161 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:18:43.676996 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39be18009970a45af27112fb42775b844e41afabb2858c74ca5f3bff170c2931\": container with ID starting with 39be18009970a45af27112fb42775b844e41afabb2858c74ca5f3bff170c2931 not found: ID does not exist" containerID="39be18009970a45af27112fb42775b844e41afabb2858c74ca5f3bff170c2931" Oct 29 23:18:43.677161 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.677031 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39be18009970a45af27112fb42775b844e41afabb2858c74ca5f3bff170c2931"} err="failed to get container status \"39be18009970a45af27112fb42775b844e41afabb2858c74ca5f3bff170c2931\": rpc error: code = NotFound desc = could not find container \"39be18009970a45af27112fb42775b844e41afabb2858c74ca5f3bff170c2931\": container with ID starting with 39be18009970a45af27112fb42775b844e41afabb2858c74ca5f3bff170c2931 not found: ID does not exist" Oct 29 23:18:43.705210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.703323 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj"] Oct 29 23:18:43.705210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.703582 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" podUID="36e06ddc-5f05-453b-831a-1d533490f290" containerName="istio-proxy" containerID="cri-o://2a89a1f048fe7456ac6d78eaac09a764c50aa613285f1c4c7985a77a368fb890" gracePeriod=30 Oct 29 23:18:43.721674 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.720076 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j"] Oct 29 23:18:43.726319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.726283 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-88j5j"] Oct 29 23:18:43.926970 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.926940 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:18:43.965090 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.965060 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:18:44.000317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.999944 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-workload-certs\") pod \"59a46795-a139-40f9-9eea-0a4f93fb244f\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " Oct 29 23:18:44.000317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:43.999992 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-workload-socket\") pod \"59a46795-a139-40f9-9eea-0a4f93fb244f\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " Oct 29 23:18:44.000317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.000075 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-podinfo\") pod \"59a46795-a139-40f9-9eea-0a4f93fb244f\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " Oct 29 23:18:44.000317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.000288 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59a46795-a139-40f9-9eea-0a4f93fb244f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59a46795-a139-40f9-9eea-0a4f93fb244f-istiod-ca-cert\") pod \"59a46795-a139-40f9-9eea-0a4f93fb244f\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " Oct 29 23:18:44.000555 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.000376 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-credential-socket\") pod \"59a46795-a139-40f9-9eea-0a4f93fb244f\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " Oct 29 23:18:44.000555 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.000439 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-token\" (UniqueName: \"kubernetes.io/projected/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-token\") pod \"59a46795-a139-40f9-9eea-0a4f93fb244f\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " Oct 29 23:18:44.000555 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.000486 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-data\") pod \"59a46795-a139-40f9-9eea-0a4f93fb244f\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " Oct 29 23:18:44.000731 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.000572 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a46795-a139-40f9-9eea-0a4f93fb244f-kube-api-access-kmv6j\" (UniqueName: \"kubernetes.io/projected/59a46795-a139-40f9-9eea-0a4f93fb244f-kube-api-access-kmv6j\") pod \"59a46795-a139-40f9-9eea-0a4f93fb244f\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " Oct 29 23:18:44.000731 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.000605 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-envoy\") pod \"59a46795-a139-40f9-9eea-0a4f93fb244f\" (UID: \"59a46795-a139-40f9-9eea-0a4f93fb244f\") " Oct 29 23:18:44.002218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.002095 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a46795-a139-40f9-9eea-0a4f93fb244f-istiod-ca-cert" pod "59a46795-a139-40f9-9eea-0a4f93fb244f" (UID: "59a46795-a139-40f9-9eea-0a4f93fb244f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:44.002462 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.002438 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-credential-socket" pod "59a46795-a139-40f9-9eea-0a4f93fb244f" (UID: "59a46795-a139-40f9-9eea-0a4f93fb244f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:44.002875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.002808 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-workload-socket" pod "59a46795-a139-40f9-9eea-0a4f93fb244f" (UID: "59a46795-a139-40f9-9eea-0a4f93fb244f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:44.002875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.002846 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-workload-certs" pod "59a46795-a139-40f9-9eea-0a4f93fb244f" (UID: "59a46795-a139-40f9-9eea-0a4f93fb244f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:44.003086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.003040 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-data" pod "59a46795-a139-40f9-9eea-0a4f93fb244f" (UID: "59a46795-a139-40f9-9eea-0a4f93fb244f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:44.005936 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.005890 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-token" pod "59a46795-a139-40f9-9eea-0a4f93fb244f" (UID: "59a46795-a139-40f9-9eea-0a4f93fb244f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:44.006946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.006903 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-podinfo" pod "59a46795-a139-40f9-9eea-0a4f93fb244f" (UID: "59a46795-a139-40f9-9eea-0a4f93fb244f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:44.007622 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.007559 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a46795-a139-40f9-9eea-0a4f93fb244f-kube-api-access-kmv6j" pod "59a46795-a139-40f9-9eea-0a4f93fb244f" (UID: "59a46795-a139-40f9-9eea-0a4f93fb244f"). InnerVolumeSpecName "kube-api-access-kmv6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:44.007840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.007818 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-envoy" pod "59a46795-a139-40f9-9eea-0a4f93fb244f" (UID: "59a46795-a139-40f9-9eea-0a4f93fb244f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:44.102884 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.102852 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-workload-socket\") pod \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " Oct 29 23:18:44.103085 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.102997 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ab845c5-deac-48c4-8a7c-b2dbada5d762-kube-api-access-zdhkc\" (UniqueName: \"kubernetes.io/projected/2ab845c5-deac-48c4-8a7c-b2dbada5d762-kube-api-access-zdhkc\") pod \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " Oct 29 23:18:44.103085 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.103028 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-envoy\") pod \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " Oct 29 23:18:44.103085 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.103047 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-token\" (UniqueName: \"kubernetes.io/projected/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-token\") pod \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " Oct 29 23:18:44.103085 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.103074 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-credential-socket\") pod \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " Oct 29 23:18:44.103350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.103109 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-podinfo\") pod \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " Oct 29 23:18:44.103350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.103146 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-workload-socket" pod "2ab845c5-deac-48c4-8a7c-b2dbada5d762" (UID: "2ab845c5-deac-48c4-8a7c-b2dbada5d762"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:44.103350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.103214 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-workload-certs\") pod \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " Oct 29 23:18:44.103350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.103252 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-data\") pod \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " Oct 29 23:18:44.103350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.103300 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istiod-ca-cert\") pod \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\" (UID: \"2ab845c5-deac-48c4-8a7c-b2dbada5d762\") " Oct 29 23:18:44.103592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.103424 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-credential-socket" pod "2ab845c5-deac-48c4-8a7c-b2dbada5d762" (UID: "2ab845c5-deac-48c4-8a7c-b2dbada5d762"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:44.103655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.103595 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-data" pod "2ab845c5-deac-48c4-8a7c-b2dbada5d762" (UID: "2ab845c5-deac-48c4-8a7c-b2dbada5d762"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:44.103770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.103726 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-workload-certs" pod "2ab845c5-deac-48c4-8a7c-b2dbada5d762" (UID: "2ab845c5-deac-48c4-8a7c-b2dbada5d762"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:44.103904 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.103884 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istiod-ca-cert" pod "2ab845c5-deac-48c4-8a7c-b2dbada5d762" (UID: "2ab845c5-deac-48c4-8a7c-b2dbada5d762"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:44.104123 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.104094 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:44.104123 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.104118 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:44.104295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.104133 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:44.104295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.104145 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:44.104295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.104157 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:44.104295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.104169 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59a46795-a139-40f9-9eea-0a4f93fb244f-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:44.104295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.104181 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:44.104295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.104229 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:44.104295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.104241 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:44.104295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.104253 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:44.104295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.104266 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmv6j\" (UniqueName: \"kubernetes.io/projected/59a46795-a139-40f9-9eea-0a4f93fb244f-kube-api-access-kmv6j\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:44.104295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.104280 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:44.104295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.104293 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:44.104654 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.104305 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59a46795-a139-40f9-9eea-0a4f93fb244f-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:44.105586 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.105562 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-envoy" pod "2ab845c5-deac-48c4-8a7c-b2dbada5d762" (UID: "2ab845c5-deac-48c4-8a7c-b2dbada5d762"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:44.105703 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.105612 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab845c5-deac-48c4-8a7c-b2dbada5d762-kube-api-access-zdhkc" pod "2ab845c5-deac-48c4-8a7c-b2dbada5d762" (UID: "2ab845c5-deac-48c4-8a7c-b2dbada5d762"). InnerVolumeSpecName "kube-api-access-zdhkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:44.106062 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.106034 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-podinfo" pod "2ab845c5-deac-48c4-8a7c-b2dbada5d762" (UID: "2ab845c5-deac-48c4-8a7c-b2dbada5d762"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:44.106133 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.106059 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-token" pod "2ab845c5-deac-48c4-8a7c-b2dbada5d762" (UID: "2ab845c5-deac-48c4-8a7c-b2dbada5d762"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:44.205584 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.205547 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdhkc\" (UniqueName: \"kubernetes.io/projected/2ab845c5-deac-48c4-8a7c-b2dbada5d762-kube-api-access-zdhkc\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:44.205584 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.205583 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:44.205793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.205595 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:44.205793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.205607 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab845c5-deac-48c4-8a7c-b2dbada5d762-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:44.618458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.617440 5057 generic.go:356] "Generic (PLEG): container finished" podID="2ab845c5-deac-48c4-8a7c-b2dbada5d762" containerID="a73459a04103a5b698341b633722ca5fa92966f8dc1f720304188a58e5c08e6e" exitCode=0 Oct 29 23:18:44.618458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.617584 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" event={"ID":"2ab845c5-deac-48c4-8a7c-b2dbada5d762","Type":"ContainerDied","Data":"a73459a04103a5b698341b633722ca5fa92966f8dc1f720304188a58e5c08e6e"} Oct 29 23:18:44.618458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.617641 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" event={"ID":"2ab845c5-deac-48c4-8a7c-b2dbada5d762","Type":"ContainerDied","Data":"8782105af1db5d00ec5cddbf03c725cbf4a1664678aba385a09788b776132740"} Oct 29 23:18:44.618458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.617679 5057 scope.go:122] "RemoveContainer" containerID="a73459a04103a5b698341b633722ca5fa92966f8dc1f720304188a58e5c08e6e" Oct 29 23:18:44.618458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.617827 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw" Oct 29 23:18:44.632006 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.631972 5057 generic.go:356] "Generic (PLEG): container finished" podID="59a46795-a139-40f9-9eea-0a4f93fb244f" containerID="0650927cf63cfd6bd31ad94bfd71b41e23b3059dbe6b629e6d2618fe6259c932" exitCode=0 Oct 29 23:18:44.632142 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.632107 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" Oct 29 23:18:44.632281 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.632182 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" event={"ID":"59a46795-a139-40f9-9eea-0a4f93fb244f","Type":"ContainerDied","Data":"0650927cf63cfd6bd31ad94bfd71b41e23b3059dbe6b629e6d2618fe6259c932"} Oct 29 23:18:44.632359 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.632316 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp" event={"ID":"59a46795-a139-40f9-9eea-0a4f93fb244f","Type":"ContainerDied","Data":"b38ff1e6fd4bfc1b42aca531ec23326d35255175b516a2294003a2d64436839a"} Oct 29 23:18:44.644846 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.644820 5057 scope.go:122] "RemoveContainer" containerID="a73459a04103a5b698341b633722ca5fa92966f8dc1f720304188a58e5c08e6e" Oct 29 23:18:44.645127 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:18:44.645106 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73459a04103a5b698341b633722ca5fa92966f8dc1f720304188a58e5c08e6e\": container with ID starting with a73459a04103a5b698341b633722ca5fa92966f8dc1f720304188a58e5c08e6e not found: ID does not exist" containerID="a73459a04103a5b698341b633722ca5fa92966f8dc1f720304188a58e5c08e6e" Oct 29 23:18:44.645224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.645136 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73459a04103a5b698341b633722ca5fa92966f8dc1f720304188a58e5c08e6e"} err="failed to get container status \"a73459a04103a5b698341b633722ca5fa92966f8dc1f720304188a58e5c08e6e\": rpc error: code = NotFound desc = could not find container \"a73459a04103a5b698341b633722ca5fa92966f8dc1f720304188a58e5c08e6e\": container with ID starting with a73459a04103a5b698341b633722ca5fa92966f8dc1f720304188a58e5c08e6e not found: ID does not exist" Oct 29 23:18:44.645224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.645161 5057 scope.go:122] "RemoveContainer" containerID="0650927cf63cfd6bd31ad94bfd71b41e23b3059dbe6b629e6d2618fe6259c932" Oct 29 23:18:44.661791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.661766 5057 scope.go:122] "RemoveContainer" containerID="0650927cf63cfd6bd31ad94bfd71b41e23b3059dbe6b629e6d2618fe6259c932" Oct 29 23:18:44.662102 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:18:44.662082 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0650927cf63cfd6bd31ad94bfd71b41e23b3059dbe6b629e6d2618fe6259c932\": container with ID starting with 0650927cf63cfd6bd31ad94bfd71b41e23b3059dbe6b629e6d2618fe6259c932 not found: ID does not exist" containerID="0650927cf63cfd6bd31ad94bfd71b41e23b3059dbe6b629e6d2618fe6259c932" Oct 29 23:18:44.662175 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.662115 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0650927cf63cfd6bd31ad94bfd71b41e23b3059dbe6b629e6d2618fe6259c932"} err="failed to get container status \"0650927cf63cfd6bd31ad94bfd71b41e23b3059dbe6b629e6d2618fe6259c932\": rpc error: code = NotFound desc = could not find container \"0650927cf63cfd6bd31ad94bfd71b41e23b3059dbe6b629e6d2618fe6259c932\": container with ID starting with 0650927cf63cfd6bd31ad94bfd71b41e23b3059dbe6b629e6d2618fe6259c932 not found: ID does not exist" Oct 29 23:18:44.682769 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.682738 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw"] Oct 29 23:18:44.695903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.695876 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-z76dw"] Oct 29 23:18:44.745230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.745198 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp"] Oct 29 23:18:44.750047 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:44.750026 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-pb7dp"] Oct 29 23:18:45.483850 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:45.483820 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ab845c5-deac-48c4-8a7c-b2dbada5d762" path="/var/lib/kubelet/pods/2ab845c5-deac-48c4-8a7c-b2dbada5d762/volumes" Oct 29 23:18:45.484444 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:45.484429 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59a46795-a139-40f9-9eea-0a4f93fb244f" path="/var/lib/kubelet/pods/59a46795-a139-40f9-9eea-0a4f93fb244f/volumes" Oct 29 23:18:45.484925 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:45.484912 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afeb2ed0-d3ab-4607-91c1-f97eb0662229" path="/var/lib/kubelet/pods/afeb2ed0-d3ab-4607-91c1-f97eb0662229/volumes" Oct 29 23:18:46.475589 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:46.475545 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:18:46.476007 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:46.475611 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" podUID="36e06ddc-5f05-453b-831a-1d533490f290" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:18:47.403511 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:47.403476 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd"] Oct 29 23:18:47.403732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:47.403693 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" podUID="326ecac6-3d7b-49ae-a584-c2b9f9aa644d" containerName="istio-proxy" containerID="cri-o://7612a42888170e20c7937b3a8a8f2d00b13643d07ae4855589ddbe6bfd2ab941" gracePeriod=30 Oct 29 23:18:49.007458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.007432 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:18:49.009683 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.009660 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36e06ddc-5f05-453b-831a-1d533490f290-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36e06ddc-5f05-453b-831a-1d533490f290-istio-podinfo\") pod \"36e06ddc-5f05-453b-831a-1d533490f290\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " Oct 29 23:18:49.009819 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.009720 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-workload-socket\") pod \"36e06ddc-5f05-453b-831a-1d533490f290\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " Oct 29 23:18:49.009819 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.009783 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36e06ddc-5f05-453b-831a-1d533490f290-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36e06ddc-5f05-453b-831a-1d533490f290-istiod-ca-cert\") pod \"36e06ddc-5f05-453b-831a-1d533490f290\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " Oct 29 23:18:49.009819 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.009806 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-workload-certs\") pod \"36e06ddc-5f05-453b-831a-1d533490f290\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " Oct 29 23:18:49.009992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.009889 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-credential-socket\") pod \"36e06ddc-5f05-453b-831a-1d533490f290\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " Oct 29 23:18:49.009992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.009932 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36e06ddc-5f05-453b-831a-1d533490f290-istio-token\" (UniqueName: \"kubernetes.io/projected/36e06ddc-5f05-453b-831a-1d533490f290-istio-token\") pod \"36e06ddc-5f05-453b-831a-1d533490f290\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " Oct 29 23:18:49.009992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.009956 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-istio-envoy\") pod \"36e06ddc-5f05-453b-831a-1d533490f290\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " Oct 29 23:18:49.009992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.009980 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36e06ddc-5f05-453b-831a-1d533490f290-kube-api-access-6r98j\" (UniqueName: \"kubernetes.io/projected/36e06ddc-5f05-453b-831a-1d533490f290-kube-api-access-6r98j\") pod \"36e06ddc-5f05-453b-831a-1d533490f290\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " Oct 29 23:18:49.010258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.010012 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-workload-socket" pod "36e06ddc-5f05-453b-831a-1d533490f290" (UID: "36e06ddc-5f05-453b-831a-1d533490f290"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:49.010258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.010117 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-istio-data\") pod \"36e06ddc-5f05-453b-831a-1d533490f290\" (UID: \"36e06ddc-5f05-453b-831a-1d533490f290\") " Oct 29 23:18:49.010258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.010147 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-workload-certs" pod "36e06ddc-5f05-453b-831a-1d533490f290" (UID: "36e06ddc-5f05-453b-831a-1d533490f290"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:49.010258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.010175 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-credential-socket" pod "36e06ddc-5f05-453b-831a-1d533490f290" (UID: "36e06ddc-5f05-453b-831a-1d533490f290"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:49.010476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.010264 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36e06ddc-5f05-453b-831a-1d533490f290-istiod-ca-cert" pod "36e06ddc-5f05-453b-831a-1d533490f290" (UID: "36e06ddc-5f05-453b-831a-1d533490f290"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:49.010476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.010432 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-istio-data" pod "36e06ddc-5f05-453b-831a-1d533490f290" (UID: "36e06ddc-5f05-453b-831a-1d533490f290"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:49.010915 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.010891 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:49.011015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.010917 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36e06ddc-5f05-453b-831a-1d533490f290-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:49.011015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.010931 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:49.011015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.010944 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:49.011015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.010957 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:49.011994 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.011971 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36e06ddc-5f05-453b-831a-1d533490f290-istio-podinfo" pod "36e06ddc-5f05-453b-831a-1d533490f290" (UID: "36e06ddc-5f05-453b-831a-1d533490f290"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:49.012170 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.012151 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e06ddc-5f05-453b-831a-1d533490f290-istio-token" pod "36e06ddc-5f05-453b-831a-1d533490f290" (UID: "36e06ddc-5f05-453b-831a-1d533490f290"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:49.012288 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.012262 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e06ddc-5f05-453b-831a-1d533490f290-kube-api-access-6r98j" pod "36e06ddc-5f05-453b-831a-1d533490f290" (UID: "36e06ddc-5f05-453b-831a-1d533490f290"). InnerVolumeSpecName "kube-api-access-6r98j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:49.012431 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.012415 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-istio-envoy" pod "36e06ddc-5f05-453b-831a-1d533490f290" (UID: "36e06ddc-5f05-453b-831a-1d533490f290"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:49.112708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.112668 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36e06ddc-5f05-453b-831a-1d533490f290-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:49.112708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.112708 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36e06ddc-5f05-453b-831a-1d533490f290-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:49.112919 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.112722 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r98j\" (UniqueName: \"kubernetes.io/projected/36e06ddc-5f05-453b-831a-1d533490f290-kube-api-access-6r98j\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:49.112919 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.112736 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36e06ddc-5f05-453b-831a-1d533490f290-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:49.672877 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.672841 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx"] Oct 29 23:18:49.673108 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.673083 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" podUID="f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" containerName="istio-proxy" containerID="cri-o://11a223d480b1391fee8a3d891876dc2fadbb7de2ab46b9ecb1fdca9b0ff1791e" gracePeriod=30 Oct 29 23:18:49.683346 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.683318 5057 generic.go:356] "Generic (PLEG): container finished" podID="36e06ddc-5f05-453b-831a-1d533490f290" containerID="2a89a1f048fe7456ac6d78eaac09a764c50aa613285f1c4c7985a77a368fb890" exitCode=0 Oct 29 23:18:49.683471 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.683424 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" Oct 29 23:18:49.683471 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.683448 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" event={"ID":"36e06ddc-5f05-453b-831a-1d533490f290","Type":"ContainerDied","Data":"2a89a1f048fe7456ac6d78eaac09a764c50aa613285f1c4c7985a77a368fb890"} Oct 29 23:18:49.683601 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.683515 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj" event={"ID":"36e06ddc-5f05-453b-831a-1d533490f290","Type":"ContainerDied","Data":"bda2f03f66e05e8fe0ea7344c3ef62e663dc8d6b478bfae360691d39e301f8d8"} Oct 29 23:18:49.683601 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.683546 5057 scope.go:122] "RemoveContainer" containerID="2a89a1f048fe7456ac6d78eaac09a764c50aa613285f1c4c7985a77a368fb890" Oct 29 23:18:49.685122 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.685102 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd"] Oct 29 23:18:49.685336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.685307 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" podUID="e794e7ae-93a6-4789-9542-217137d0aa1a" containerName="istio-proxy" containerID="cri-o://9c7ee8762d549d259c59d7f7f1b74f5eb9bc58dbdc0472a790e08a3809a2dacc" gracePeriod=30 Oct 29 23:18:49.690056 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.690037 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp"] Oct 29 23:18:49.690742 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.690224 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" podUID="503f6577-e8f6-474e-8b57-98e30edaa785" containerName="istio-proxy" containerID="cri-o://d9096ca87b012ca5371d44c49d9e3ba229133e1217ec96b2df51d0fcca9b52da" gracePeriod=30 Oct 29 23:18:49.696834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.696810 5057 scope.go:122] "RemoveContainer" containerID="2a89a1f048fe7456ac6d78eaac09a764c50aa613285f1c4c7985a77a368fb890" Oct 29 23:18:49.697134 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:18:49.697115 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a89a1f048fe7456ac6d78eaac09a764c50aa613285f1c4c7985a77a368fb890\": container with ID starting with 2a89a1f048fe7456ac6d78eaac09a764c50aa613285f1c4c7985a77a368fb890 not found: ID does not exist" containerID="2a89a1f048fe7456ac6d78eaac09a764c50aa613285f1c4c7985a77a368fb890" Oct 29 23:18:49.697245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.697141 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a89a1f048fe7456ac6d78eaac09a764c50aa613285f1c4c7985a77a368fb890"} err="failed to get container status \"2a89a1f048fe7456ac6d78eaac09a764c50aa613285f1c4c7985a77a368fb890\": rpc error: code = NotFound desc = could not find container \"2a89a1f048fe7456ac6d78eaac09a764c50aa613285f1c4c7985a77a368fb890\": container with ID starting with 2a89a1f048fe7456ac6d78eaac09a764c50aa613285f1c4c7985a77a368fb890 not found: ID does not exist" Oct 29 23:18:49.697801 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.697784 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf"] Oct 29 23:18:49.697949 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.697933 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" podUID="7ea22168-c52a-43f3-a4b6-4cdd87cffdae" containerName="istio-proxy" containerID="cri-o://bf1123dde721ceff0320388d9b207cd4547212466dd22e13ddf134c9ca522cde" gracePeriod=30 Oct 29 23:18:49.944914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.944834 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj"] Oct 29 23:18:49.950611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:49.950067 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-glwkj"] Oct 29 23:18:51.484879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:51.484840 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36e06ddc-5f05-453b-831a-1d533490f290" path="/var/lib/kubelet/pods/36e06ddc-5f05-453b-831a-1d533490f290/volumes" Oct 29 23:18:51.662447 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:51.662407 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:18:51.662760 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:51.662730 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" podUID="326ecac6-3d7b-49ae-a584-c2b9f9aa644d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:18:52.711113 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.711084 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 23:18:52.722492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.722444 5057 generic.go:356] "Generic (PLEG): container finished" podID="326ecac6-3d7b-49ae-a584-c2b9f9aa644d" containerID="7612a42888170e20c7937b3a8a8f2d00b13643d07ae4855589ddbe6bfd2ab941" exitCode=0 Oct 29 23:18:52.722678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.722543 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" Oct 29 23:18:52.722678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.722560 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" event={"ID":"326ecac6-3d7b-49ae-a584-c2b9f9aa644d","Type":"ContainerDied","Data":"7612a42888170e20c7937b3a8a8f2d00b13643d07ae4855589ddbe6bfd2ab941"} Oct 29 23:18:52.722678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.722608 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd" event={"ID":"326ecac6-3d7b-49ae-a584-c2b9f9aa644d","Type":"ContainerDied","Data":"bf185106cdef51aa393dd1b6ee83b57ed2b45684271213e8c5b409c11db34940"} Oct 29 23:18:52.722678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.722651 5057 scope.go:122] "RemoveContainer" containerID="7612a42888170e20c7937b3a8a8f2d00b13643d07ae4855589ddbe6bfd2ab941" Oct 29 23:18:52.733951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.733930 5057 scope.go:122] "RemoveContainer" containerID="d0ee5a238ac445666cc183f70c1329faeffd223a838497fb420aaa17c6b5ff31" Oct 29 23:18:52.746390 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.746362 5057 scope.go:122] "RemoveContainer" containerID="7612a42888170e20c7937b3a8a8f2d00b13643d07ae4855589ddbe6bfd2ab941" Oct 29 23:18:52.746677 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:18:52.746654 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7612a42888170e20c7937b3a8a8f2d00b13643d07ae4855589ddbe6bfd2ab941\": container with ID starting with 7612a42888170e20c7937b3a8a8f2d00b13643d07ae4855589ddbe6bfd2ab941 not found: ID does not exist" containerID="7612a42888170e20c7937b3a8a8f2d00b13643d07ae4855589ddbe6bfd2ab941" Oct 29 23:18:52.746765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.746684 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7612a42888170e20c7937b3a8a8f2d00b13643d07ae4855589ddbe6bfd2ab941"} err="failed to get container status \"7612a42888170e20c7937b3a8a8f2d00b13643d07ae4855589ddbe6bfd2ab941\": rpc error: code = NotFound desc = could not find container \"7612a42888170e20c7937b3a8a8f2d00b13643d07ae4855589ddbe6bfd2ab941\": container with ID starting with 7612a42888170e20c7937b3a8a8f2d00b13643d07ae4855589ddbe6bfd2ab941 not found: ID does not exist" Oct 29 23:18:52.746765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.746703 5057 scope.go:122] "RemoveContainer" containerID="d0ee5a238ac445666cc183f70c1329faeffd223a838497fb420aaa17c6b5ff31" Oct 29 23:18:52.746970 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:18:52.746949 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0ee5a238ac445666cc183f70c1329faeffd223a838497fb420aaa17c6b5ff31\": container with ID starting with d0ee5a238ac445666cc183f70c1329faeffd223a838497fb420aaa17c6b5ff31 not found: ID does not exist" containerID="d0ee5a238ac445666cc183f70c1329faeffd223a838497fb420aaa17c6b5ff31" Oct 29 23:18:52.747026 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.746979 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ee5a238ac445666cc183f70c1329faeffd223a838497fb420aaa17c6b5ff31"} err="failed to get container status \"d0ee5a238ac445666cc183f70c1329faeffd223a838497fb420aaa17c6b5ff31\": rpc error: code = NotFound desc = could not find container \"d0ee5a238ac445666cc183f70c1329faeffd223a838497fb420aaa17c6b5ff31\": container with ID starting with d0ee5a238ac445666cc183f70c1329faeffd223a838497fb420aaa17c6b5ff31 not found: ID does not exist" Oct 29 23:18:52.798886 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.798855 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istiod-ca-cert\") pod \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " Oct 29 23:18:52.799056 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.798943 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-envoy\") pod \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " Oct 29 23:18:52.799056 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.798974 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-credential-socket\") pod \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " Oct 29 23:18:52.799056 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.799051 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-token\" (UniqueName: \"kubernetes.io/projected/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-token\") pod \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " Oct 29 23:18:52.799242 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.799091 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-workload-socket\") pod \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " Oct 29 23:18:52.799242 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.799126 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-podinfo\") pod \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " Oct 29 23:18:52.799242 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.799158 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-data\") pod \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " Oct 29 23:18:52.799380 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.799274 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-workload-certs\") pod \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " Oct 29 23:18:52.799380 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.799324 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istiod-ca-cert" pod "326ecac6-3d7b-49ae-a584-c2b9f9aa644d" (UID: "326ecac6-3d7b-49ae-a584-c2b9f9aa644d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:52.799484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.799407 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-kube-api-access-9bm7x\" (UniqueName: \"kubernetes.io/projected/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-kube-api-access-9bm7x\") pod \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\" (UID: \"326ecac6-3d7b-49ae-a584-c2b9f9aa644d\") " Oct 29 23:18:52.799617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.799598 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-credential-socket" pod "326ecac6-3d7b-49ae-a584-c2b9f9aa644d" (UID: "326ecac6-3d7b-49ae-a584-c2b9f9aa644d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:52.799706 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.799683 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-workload-socket" pod "326ecac6-3d7b-49ae-a584-c2b9f9aa644d" (UID: "326ecac6-3d7b-49ae-a584-c2b9f9aa644d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:52.799802 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.799785 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-workload-certs" pod "326ecac6-3d7b-49ae-a584-c2b9f9aa644d" (UID: "326ecac6-3d7b-49ae-a584-c2b9f9aa644d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:52.800326 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.800247 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:52.800326 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.800275 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:52.800326 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.800289 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:52.800326 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.800303 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:52.802265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.802238 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-kube-api-access-9bm7x" pod "326ecac6-3d7b-49ae-a584-c2b9f9aa644d" (UID: "326ecac6-3d7b-49ae-a584-c2b9f9aa644d"). InnerVolumeSpecName "kube-api-access-9bm7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:52.802446 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.802316 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-podinfo" pod "326ecac6-3d7b-49ae-a584-c2b9f9aa644d" (UID: "326ecac6-3d7b-49ae-a584-c2b9f9aa644d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:52.802542 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.802522 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-token" pod "326ecac6-3d7b-49ae-a584-c2b9f9aa644d" (UID: "326ecac6-3d7b-49ae-a584-c2b9f9aa644d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:52.802932 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.802912 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-envoy" pod "326ecac6-3d7b-49ae-a584-c2b9f9aa644d" (UID: "326ecac6-3d7b-49ae-a584-c2b9f9aa644d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:52.803427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.803404 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-data" pod "326ecac6-3d7b-49ae-a584-c2b9f9aa644d" (UID: "326ecac6-3d7b-49ae-a584-c2b9f9aa644d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:52.901955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.901915 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:52.901955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.901954 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:52.902347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.901968 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:52.902347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.901980 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:52.902347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:52.901994 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bm7x\" (UniqueName: \"kubernetes.io/projected/326ecac6-3d7b-49ae-a584-c2b9f9aa644d-kube-api-access-9bm7x\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:53.078831 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.078792 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd"] Oct 29 23:18:53.083500 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.083469 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-t6wmd"] Oct 29 23:18:53.377510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.377433 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh"] Oct 29 23:18:53.378238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.378217 5057 cpu_manager.go:414] "containerMap: removing container" podUID="afeb2ed0-d3ab-4607-91c1-f97eb0662229" containerName="istio-proxy" Oct 29 23:18:53.378238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.378234 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afeb2ed0-d3ab-4607-91c1-f97eb0662229" containerName="istio-proxy" Oct 29 23:18:53.378433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.378248 5057 cpu_manager.go:414] "containerMap: removing container" podUID="326ecac6-3d7b-49ae-a584-c2b9f9aa644d" containerName="istio-proxy" Oct 29 23:18:53.378433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.378253 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="326ecac6-3d7b-49ae-a584-c2b9f9aa644d" containerName="istio-proxy" Oct 29 23:18:53.378433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.378266 5057 cpu_manager.go:414] "containerMap: removing container" podUID="36e06ddc-5f05-453b-831a-1d533490f290" containerName="istio-proxy" Oct 29 23:18:53.378433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.378272 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36e06ddc-5f05-453b-831a-1d533490f290" containerName="istio-proxy" Oct 29 23:18:53.378433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.378307 5057 cpu_manager.go:414] "containerMap: removing container" podUID="2ab845c5-deac-48c4-8a7c-b2dbada5d762" containerName="istio-proxy" Oct 29 23:18:53.378433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.378312 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab845c5-deac-48c4-8a7c-b2dbada5d762" containerName="istio-proxy" Oct 29 23:18:53.378433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.378319 5057 cpu_manager.go:414] "containerMap: removing container" podUID="59a46795-a139-40f9-9eea-0a4f93fb244f" containerName="istio-proxy" Oct 29 23:18:53.378433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.378323 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a46795-a139-40f9-9eea-0a4f93fb244f" containerName="istio-proxy" Oct 29 23:18:53.378433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.378337 5057 cpu_manager.go:414] "containerMap: removing container" podUID="326ecac6-3d7b-49ae-a584-c2b9f9aa644d" containerName="istio-proxy" Oct 29 23:18:53.378433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.378342 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="326ecac6-3d7b-49ae-a584-c2b9f9aa644d" containerName="istio-proxy" Oct 29 23:18:53.378975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.378458 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="326ecac6-3d7b-49ae-a584-c2b9f9aa644d" containerName="istio-proxy" Oct 29 23:18:53.378975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.378470 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="afeb2ed0-d3ab-4607-91c1-f97eb0662229" containerName="istio-proxy" Oct 29 23:18:53.378975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.378477 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="326ecac6-3d7b-49ae-a584-c2b9f9aa644d" containerName="istio-proxy" Oct 29 23:18:53.378975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.378484 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="36e06ddc-5f05-453b-831a-1d533490f290" containerName="istio-proxy" Oct 29 23:18:53.378975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.378495 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab845c5-deac-48c4-8a7c-b2dbada5d762" containerName="istio-proxy" Oct 29 23:18:53.378975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.378501 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a46795-a139-40f9-9eea-0a4f93fb244f" containerName="istio-proxy" Oct 29 23:18:53.384110 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.384087 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.403538 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.403509 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-26b322-openshift-default-dockercfg-4kj6k\"" Oct 29 23:18:53.409938 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.409893 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh"] Oct 29 23:18:53.411583 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.411555 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-workload-socket\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.411687 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.411596 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-istio-data\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.411747 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.411721 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d89378ad-ddab-4899-9041-7c4cabafa109-istio-podinfo\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.411792 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.411761 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-credential-socket\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.411792 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.411784 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-workload-certs\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.411873 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.411805 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvp4w\" (UniqueName: \"kubernetes.io/projected/d89378ad-ddab-4899-9041-7c4cabafa109-kube-api-access-mvp4w\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.411941 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.411927 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d89378ad-ddab-4899-9041-7c4cabafa109-istiod-ca-cert\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.412003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.411991 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-istio-envoy\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.412042 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.412009 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d89378ad-ddab-4899-9041-7c4cabafa109-istio-token\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.541765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.539751 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-istio-envoy\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.541765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.539799 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d89378ad-ddab-4899-9041-7c4cabafa109-istio-token\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.541765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.539903 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-workload-socket\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.541765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.539932 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-istio-data\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.541765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.540054 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d89378ad-ddab-4899-9041-7c4cabafa109-istio-podinfo\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.541765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.540106 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-credential-socket\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.541765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.540138 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-workload-certs\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.541765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.540172 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvp4w\" (UniqueName: \"kubernetes.io/projected/d89378ad-ddab-4899-9041-7c4cabafa109-kube-api-access-mvp4w\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.541765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.540451 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d89378ad-ddab-4899-9041-7c4cabafa109-istiod-ca-cert\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.541765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.541387 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d89378ad-ddab-4899-9041-7c4cabafa109-istiod-ca-cert\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.568107 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.566123 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-workload-socket\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.568107 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.566676 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-workload-certs\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.568107 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.566907 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-credential-socket\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.568107 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.567130 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-istio-data\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.645750 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.645621 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvp4w\" (UniqueName: \"kubernetes.io/projected/d89378ad-ddab-4899-9041-7c4cabafa109-kube-api-access-mvp4w\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.670927 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.669784 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d89378ad-ddab-4899-9041-7c4cabafa109-istio-token\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.670927 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.670039 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-istio-envoy\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.672024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.671987 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d89378ad-ddab-4899-9041-7c4cabafa109-istio-podinfo\") pod \"e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.715302 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.715007 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:53.739308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:53.722900 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="326ecac6-3d7b-49ae-a584-c2b9f9aa644d" path="/var/lib/kubelet/pods/326ecac6-3d7b-49ae-a584-c2b9f9aa644d/volumes" Oct 29 23:18:54.331317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:54.331283 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh"] Oct 29 23:18:54.331458 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:18:54.331356 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89378ad_ddab_4899_9041_7c4cabafa109.slice/crio-39bd9e81c0d191f8de486c0e62854923b9472a1f2c92c4473003244a0c9b36d6 WatchSource:0}: Error finding container 39bd9e81c0d191f8de486c0e62854923b9472a1f2c92c4473003244a0c9b36d6: Status 404 returned error can't find the container with id 39bd9e81c0d191f8de486c0e62854923b9472a1f2c92c4473003244a0c9b36d6 Oct 29 23:18:54.334746 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:54.334672 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:18:54.334835 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:54.334788 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:18:54.334835 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:54.334829 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:18:54.895282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:54.895244 5057 generic.go:356] "Generic (PLEG): container finished" podID="503f6577-e8f6-474e-8b57-98e30edaa785" containerID="d9096ca87b012ca5371d44c49d9e3ba229133e1217ec96b2df51d0fcca9b52da" exitCode=0 Oct 29 23:18:54.895699 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:54.895342 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" event={"ID":"503f6577-e8f6-474e-8b57-98e30edaa785","Type":"ContainerDied","Data":"d9096ca87b012ca5371d44c49d9e3ba229133e1217ec96b2df51d0fcca9b52da"} Oct 29 23:18:54.898116 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:54.897848 5057 generic.go:356] "Generic (PLEG): container finished" podID="f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" containerID="11a223d480b1391fee8a3d891876dc2fadbb7de2ab46b9ecb1fdca9b0ff1791e" exitCode=0 Oct 29 23:18:54.898116 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:54.898005 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" event={"ID":"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a","Type":"ContainerDied","Data":"11a223d480b1391fee8a3d891876dc2fadbb7de2ab46b9ecb1fdca9b0ff1791e"} Oct 29 23:18:54.900500 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:54.900176 5057 generic.go:356] "Generic (PLEG): container finished" podID="e794e7ae-93a6-4789-9542-217137d0aa1a" containerID="9c7ee8762d549d259c59d7f7f1b74f5eb9bc58dbdc0472a790e08a3809a2dacc" exitCode=0 Oct 29 23:18:54.900708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:54.900639 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" event={"ID":"e794e7ae-93a6-4789-9542-217137d0aa1a","Type":"ContainerDied","Data":"9c7ee8762d549d259c59d7f7f1b74f5eb9bc58dbdc0472a790e08a3809a2dacc"} Oct 29 23:18:54.902946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:54.902740 5057 generic.go:356] "Generic (PLEG): container finished" podID="7ea22168-c52a-43f3-a4b6-4cdd87cffdae" containerID="bf1123dde721ceff0320388d9b207cd4547212466dd22e13ddf134c9ca522cde" exitCode=0 Oct 29 23:18:54.903136 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:54.903072 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" event={"ID":"7ea22168-c52a-43f3-a4b6-4cdd87cffdae","Type":"ContainerDied","Data":"bf1123dde721ceff0320388d9b207cd4547212466dd22e13ddf134c9ca522cde"} Oct 29 23:18:54.906614 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:54.905600 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" event={"ID":"d89378ad-ddab-4899-9041-7c4cabafa109","Type":"ContainerStarted","Data":"38f1f62e9e3c5f520f43a26031fde07bebe829a1260e36b8ac12b46fb3d650d0"} Oct 29 23:18:54.906614 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:54.905646 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" event={"ID":"d89378ad-ddab-4899-9041-7c4cabafa109","Type":"ContainerStarted","Data":"39bd9e81c0d191f8de486c0e62854923b9472a1f2c92c4473003244a0c9b36d6"} Oct 29 23:18:55.001070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.000962 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" podStartSLOduration=2.000942527 podStartE2EDuration="2.000942527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:54.998964103 +0000 UTC m=+2181.650462642" watchObservedRunningTime="2026-10-29 23:18:55.000942527 +0000 UTC m=+2181.652441074" Oct 29 23:18:55.088381 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.088290 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 23:18:55.099136 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.099111 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:18:55.102428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.102406 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.239012 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-data\") pod \"e794e7ae-93a6-4789-9542-217137d0aa1a\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.239067 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-kube-api-access-74sdl\" (UniqueName: \"kubernetes.io/projected/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-kube-api-access-74sdl\") pod \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.239124 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503f6577-e8f6-474e-8b57-98e30edaa785-istio-token\" (UniqueName: \"kubernetes.io/projected/503f6577-e8f6-474e-8b57-98e30edaa785-istio-token\") pod \"503f6577-e8f6-474e-8b57-98e30edaa785\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.239152 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-token\" (UniqueName: \"kubernetes.io/projected/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-token\") pod \"e794e7ae-93a6-4789-9542-217137d0aa1a\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.239181 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/503f6577-e8f6-474e-8b57-98e30edaa785-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503f6577-e8f6-474e-8b57-98e30edaa785-istio-podinfo\") pod \"503f6577-e8f6-474e-8b57-98e30edaa785\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.239253 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-credential-socket\") pod \"503f6577-e8f6-474e-8b57-98e30edaa785\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.239303 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istiod-ca-cert\") pod \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.239349 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-workload-certs\") pod \"e794e7ae-93a6-4789-9542-217137d0aa1a\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.239391 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e794e7ae-93a6-4789-9542-217137d0aa1a-kube-api-access-lwfg9\" (UniqueName: \"kubernetes.io/projected/e794e7ae-93a6-4789-9542-217137d0aa1a-kube-api-access-lwfg9\") pod \"e794e7ae-93a6-4789-9542-217137d0aa1a\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.239493 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-podinfo\") pod \"e794e7ae-93a6-4789-9542-217137d0aa1a\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.239610 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-workload-socket\") pod \"e794e7ae-93a6-4789-9542-217137d0aa1a\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.239646 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-credential-socket\") pod \"e794e7ae-93a6-4789-9542-217137d0aa1a\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.239726 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-istio-data\") pod \"503f6577-e8f6-474e-8b57-98e30edaa785\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.239778 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-credential-socket\") pod \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.239808 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/503f6577-e8f6-474e-8b57-98e30edaa785-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503f6577-e8f6-474e-8b57-98e30edaa785-istiod-ca-cert\") pod \"503f6577-e8f6-474e-8b57-98e30edaa785\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.239843 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-workload-certs\") pod \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.239870 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-token\" (UniqueName: \"kubernetes.io/projected/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-token\") pod \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.239934 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-workload-socket\") pod \"503f6577-e8f6-474e-8b57-98e30edaa785\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.239969 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-workload-socket\") pod \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.240007 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-envoy\") pod \"e794e7ae-93a6-4789-9542-217137d0aa1a\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.240034 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-istio-envoy\") pod \"503f6577-e8f6-474e-8b57-98e30edaa785\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.240105 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-podinfo\") pod \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.240148 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-workload-certs\") pod \"503f6577-e8f6-474e-8b57-98e30edaa785\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.240283 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-envoy\") pod \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.240327 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-data\") pod \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\" (UID: \"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.240357 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e794e7ae-93a6-4789-9542-217137d0aa1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e794e7ae-93a6-4789-9542-217137d0aa1a-istiod-ca-cert\") pod \"e794e7ae-93a6-4789-9542-217137d0aa1a\" (UID: \"e794e7ae-93a6-4789-9542-217137d0aa1a\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.240420 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503f6577-e8f6-474e-8b57-98e30edaa785-kube-api-access-627wt\" (UniqueName: \"kubernetes.io/projected/503f6577-e8f6-474e-8b57-98e30edaa785-kube-api-access-627wt\") pod \"503f6577-e8f6-474e-8b57-98e30edaa785\" (UID: \"503f6577-e8f6-474e-8b57-98e30edaa785\") " Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.257019 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503f6577-e8f6-474e-8b57-98e30edaa785-kube-api-access-627wt" pod "503f6577-e8f6-474e-8b57-98e30edaa785" (UID: "503f6577-e8f6-474e-8b57-98e30edaa785"). InnerVolumeSpecName "kube-api-access-627wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.257448 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-credential-socket" pod "f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" (UID: "f2a9c05b-ea5f-4f38-8043-a3ed71754e1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.257744 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/503f6577-e8f6-474e-8b57-98e30edaa785-istiod-ca-cert" pod "503f6577-e8f6-474e-8b57-98e30edaa785" (UID: "503f6577-e8f6-474e-8b57-98e30edaa785"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.258055 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-workload-certs" pod "f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" (UID: "f2a9c05b-ea5f-4f38-8043-a3ed71754e1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.261373 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-istio-data" pod "503f6577-e8f6-474e-8b57-98e30edaa785" (UID: "503f6577-e8f6-474e-8b57-98e30edaa785"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.264605 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istiod-ca-cert" pod "f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" (UID: "f2a9c05b-ea5f-4f38-8043-a3ed71754e1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.264907 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-workload-certs" pod "e794e7ae-93a6-4789-9542-217137d0aa1a" (UID: "e794e7ae-93a6-4789-9542-217137d0aa1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.276284 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-workload-certs" pod "503f6577-e8f6-474e-8b57-98e30edaa785" (UID: "503f6577-e8f6-474e-8b57-98e30edaa785"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.276616 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-credential-socket" pod "503f6577-e8f6-474e-8b57-98e30edaa785" (UID: "503f6577-e8f6-474e-8b57-98e30edaa785"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.288842 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-token" pod "f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" (UID: "f2a9c05b-ea5f-4f38-8043-a3ed71754e1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.294436 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-data" pod "e794e7ae-93a6-4789-9542-217137d0aa1a" (UID: "e794e7ae-93a6-4789-9542-217137d0aa1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.296117 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-data" pod "f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" (UID: "f2a9c05b-ea5f-4f38-8043-a3ed71754e1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.297466 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-workload-socket" pod "503f6577-e8f6-474e-8b57-98e30edaa785" (UID: "503f6577-e8f6-474e-8b57-98e30edaa785"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.297665 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-workload-socket" pod "f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" (UID: "f2a9c05b-ea5f-4f38-8043-a3ed71754e1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.299519 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-workload-socket" pod "e794e7ae-93a6-4789-9542-217137d0aa1a" (UID: "e794e7ae-93a6-4789-9542-217137d0aa1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.310502 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-credential-socket" pod "e794e7ae-93a6-4789-9542-217137d0aa1a" (UID: "e794e7ae-93a6-4789-9542-217137d0aa1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.321095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.311390 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e794e7ae-93a6-4789-9542-217137d0aa1a-istiod-ca-cert" pod "e794e7ae-93a6-4789-9542-217137d0aa1a" (UID: "e794e7ae-93a6-4789-9542-217137d0aa1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:55.350135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.346970 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.350135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.347004 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.350135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.347018 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.350135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.347031 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.350135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.347044 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503f6577-e8f6-474e-8b57-98e30edaa785-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.350135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.347059 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.350135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.347072 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.350135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.347085 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.350135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.347097 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.350135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.347109 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.350135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.347121 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.350135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.347132 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e794e7ae-93a6-4789-9542-217137d0aa1a-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.350135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.347145 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-627wt\" (UniqueName: \"kubernetes.io/projected/503f6577-e8f6-474e-8b57-98e30edaa785-kube-api-access-627wt\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.350135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.347157 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.350135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.347172 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.350135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.347202 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.350135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.347217 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.350135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.349372 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-token" pod "e794e7ae-93a6-4789-9542-217137d0aa1a" (UID: "e794e7ae-93a6-4789-9542-217137d0aa1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:55.350135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.349450 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-podinfo" pod "e794e7ae-93a6-4789-9542-217137d0aa1a" (UID: "e794e7ae-93a6-4789-9542-217137d0aa1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:55.350135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.349513 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-podinfo" pod "f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" (UID: "f2a9c05b-ea5f-4f38-8043-a3ed71754e1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:55.351944 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.351344 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-envoy" pod "e794e7ae-93a6-4789-9542-217137d0aa1a" (UID: "e794e7ae-93a6-4789-9542-217137d0aa1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.351944 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.351452 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503f6577-e8f6-474e-8b57-98e30edaa785-istio-token" pod "503f6577-e8f6-474e-8b57-98e30edaa785" (UID: "503f6577-e8f6-474e-8b57-98e30edaa785"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:55.351944 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.351542 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e794e7ae-93a6-4789-9542-217137d0aa1a-kube-api-access-lwfg9" pod "e794e7ae-93a6-4789-9542-217137d0aa1a" (UID: "e794e7ae-93a6-4789-9542-217137d0aa1a"). InnerVolumeSpecName "kube-api-access-lwfg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:55.352329 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.352219 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/503f6577-e8f6-474e-8b57-98e30edaa785-istio-podinfo" pod "503f6577-e8f6-474e-8b57-98e30edaa785" (UID: "503f6577-e8f6-474e-8b57-98e30edaa785"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:55.352329 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.352289 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-istio-envoy" pod "503f6577-e8f6-474e-8b57-98e30edaa785" (UID: "503f6577-e8f6-474e-8b57-98e30edaa785"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.353336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.353297 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-kube-api-access-74sdl" pod "f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" (UID: "f2a9c05b-ea5f-4f38-8043-a3ed71754e1a"). InnerVolumeSpecName "kube-api-access-74sdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:55.360499 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.360446 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-envoy" pod "f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" (UID: "f2a9c05b-ea5f-4f38-8043-a3ed71754e1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.414029 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.414001 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:18:55.449353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.449322 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.449353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.449350 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503f6577-e8f6-474e-8b57-98e30edaa785-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.449353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.449359 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.449591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.449368 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.449591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.449389 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74sdl\" (UniqueName: \"kubernetes.io/projected/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a-kube-api-access-74sdl\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.449591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.449398 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/503f6577-e8f6-474e-8b57-98e30edaa785-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.449591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.449408 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.449591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.449415 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503f6577-e8f6-474e-8b57-98e30edaa785-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.449591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.449423 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwfg9\" (UniqueName: \"kubernetes.io/projected/e794e7ae-93a6-4789-9542-217137d0aa1a-kube-api-access-lwfg9\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.449591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.449431 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e794e7ae-93a6-4789-9542-217137d0aa1a-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.550913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.550871 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-podinfo\") pod \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " Oct 29 23:18:55.551175 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.550937 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-data\") pod \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " Oct 29 23:18:55.551175 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.550980 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-token\" (UniqueName: \"kubernetes.io/projected/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-token\") pod \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " Oct 29 23:18:55.551175 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.551061 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-workload-certs\") pod \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " Oct 29 23:18:55.551175 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.551103 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-kube-api-access-xwdbj\" (UniqueName: \"kubernetes.io/projected/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-kube-api-access-xwdbj\") pod \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " Oct 29 23:18:55.551491 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.551214 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istiod-ca-cert\") pod \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " Oct 29 23:18:55.551491 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.551467 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-workload-certs" pod "7ea22168-c52a-43f3-a4b6-4cdd87cffdae" (UID: "7ea22168-c52a-43f3-a4b6-4cdd87cffdae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.551622 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.551479 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-credential-socket\") pod \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " Oct 29 23:18:55.551622 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.551573 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-envoy\") pod \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " Oct 29 23:18:55.551729 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.551629 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-workload-socket\") pod \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\" (UID: \"7ea22168-c52a-43f3-a4b6-4cdd87cffdae\") " Oct 29 23:18:55.551729 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.551709 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-credential-socket" pod "7ea22168-c52a-43f3-a4b6-4cdd87cffdae" (UID: "7ea22168-c52a-43f3-a4b6-4cdd87cffdae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.552135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.552015 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istiod-ca-cert" pod "7ea22168-c52a-43f3-a4b6-4cdd87cffdae" (UID: "7ea22168-c52a-43f3-a4b6-4cdd87cffdae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:55.552290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.552139 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-workload-socket" pod "7ea22168-c52a-43f3-a4b6-4cdd87cffdae" (UID: "7ea22168-c52a-43f3-a4b6-4cdd87cffdae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.552290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.552220 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-data" pod "7ea22168-c52a-43f3-a4b6-4cdd87cffdae" (UID: "7ea22168-c52a-43f3-a4b6-4cdd87cffdae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.552991 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.552899 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.552991 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.552928 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.552991 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.552943 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.552991 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.552957 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.552991 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.552970 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.554495 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.554475 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-kube-api-access-xwdbj" pod "7ea22168-c52a-43f3-a4b6-4cdd87cffdae" (UID: "7ea22168-c52a-43f3-a4b6-4cdd87cffdae"). InnerVolumeSpecName "kube-api-access-xwdbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:55.554628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.554614 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-podinfo" pod "7ea22168-c52a-43f3-a4b6-4cdd87cffdae" (UID: "7ea22168-c52a-43f3-a4b6-4cdd87cffdae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:55.554811 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.554790 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-token" pod "7ea22168-c52a-43f3-a4b6-4cdd87cffdae" (UID: "7ea22168-c52a-43f3-a4b6-4cdd87cffdae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:55.554954 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.554932 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-envoy" pod "7ea22168-c52a-43f3-a4b6-4cdd87cffdae" (UID: "7ea22168-c52a-43f3-a4b6-4cdd87cffdae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.654834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.654804 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.654834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.654833 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.654834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.654848 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.655142 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.654861 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwdbj\" (UniqueName: \"kubernetes.io/projected/7ea22168-c52a-43f3-a4b6-4cdd87cffdae-kube-api-access-xwdbj\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.716098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.716064 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:55.722359 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.722326 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:55.915996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.915918 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" Oct 29 23:18:55.916506 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.915923 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp" event={"ID":"503f6577-e8f6-474e-8b57-98e30edaa785","Type":"ContainerDied","Data":"60e83972324790e6433edddb6530c63ee86988c54ddb053a64c6c86b41ada010"} Oct 29 23:18:55.916506 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.916061 5057 scope.go:122] "RemoveContainer" containerID="d9096ca87b012ca5371d44c49d9e3ba229133e1217ec96b2df51d0fcca9b52da" Oct 29 23:18:55.918088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.917477 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" Oct 29 23:18:55.918088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.917504 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx" event={"ID":"f2a9c05b-ea5f-4f38-8043-a3ed71754e1a","Type":"ContainerDied","Data":"cadccce3d371c87a2799e6f4cf0d0168fc499fc413990d67903dd48b383df317"} Oct 29 23:18:55.919322 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.919292 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" Oct 29 23:18:55.919427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.919296 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd" event={"ID":"e794e7ae-93a6-4789-9542-217137d0aa1a","Type":"ContainerDied","Data":"164f036696e270d5516c3112360e8fbad396217da5645220302287c0f3c3fac3"} Oct 29 23:18:55.921130 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.921110 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" Oct 29 23:18:55.921277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.921146 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf" event={"ID":"7ea22168-c52a-43f3-a4b6-4cdd87cffdae","Type":"ContainerDied","Data":"a3a7eea2a268a5aaa1d5da4573466a51a593973a2174197d664f2e071ec6eaa7"} Oct 29 23:18:55.921576 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.921490 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:55.922762 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.922741 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:18:55.932245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.932227 5057 scope.go:122] "RemoveContainer" containerID="11a223d480b1391fee8a3d891876dc2fadbb7de2ab46b9ecb1fdca9b0ff1791e" Oct 29 23:18:55.942619 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.942599 5057 scope.go:122] "RemoveContainer" containerID="9c7ee8762d549d259c59d7f7f1b74f5eb9bc58dbdc0472a790e08a3809a2dacc" Oct 29 23:18:55.953457 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:55.953432 5057 scope.go:122] "RemoveContainer" containerID="bf1123dde721ceff0320388d9b207cd4547212466dd22e13ddf134c9ca522cde" Oct 29 23:18:56.046213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:56.046162 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd"] Oct 29 23:18:56.055003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:56.054973 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-tprxd"] Oct 29 23:18:56.083746 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:56.083719 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx"] Oct 29 23:18:56.092277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:56.092245 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-ffnvx"] Oct 29 23:18:56.126817 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:56.126783 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf"] Oct 29 23:18:56.131126 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:56.131097 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xvpdf"] Oct 29 23:18:56.168013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:56.167947 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp"] Oct 29 23:18:56.173026 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:56.173002 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-5kjpp"] Oct 29 23:18:57.484357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:57.484324 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="503f6577-e8f6-474e-8b57-98e30edaa785" path="/var/lib/kubelet/pods/503f6577-e8f6-474e-8b57-98e30edaa785/volumes" Oct 29 23:18:57.485016 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:57.484996 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ea22168-c52a-43f3-a4b6-4cdd87cffdae" path="/var/lib/kubelet/pods/7ea22168-c52a-43f3-a4b6-4cdd87cffdae/volumes" Oct 29 23:18:57.485551 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:57.485536 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e794e7ae-93a6-4789-9542-217137d0aa1a" path="/var/lib/kubelet/pods/e794e7ae-93a6-4789-9542-217137d0aa1a/volumes" Oct 29 23:18:57.486354 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:57.486339 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" path="/var/lib/kubelet/pods/f2a9c05b-ea5f-4f38-8043-a3ed71754e1a/volumes" Oct 29 23:18:59.267777 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.267743 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs"] Oct 29 23:18:59.268887 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.268864 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" containerName="istio-proxy" Oct 29 23:18:59.268963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.268891 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" containerName="istio-proxy" Oct 29 23:18:59.268963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.268936 5057 cpu_manager.go:414] "containerMap: removing container" podUID="503f6577-e8f6-474e-8b57-98e30edaa785" containerName="istio-proxy" Oct 29 23:18:59.268963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.268943 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="503f6577-e8f6-474e-8b57-98e30edaa785" containerName="istio-proxy" Oct 29 23:18:59.269064 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.268972 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e794e7ae-93a6-4789-9542-217137d0aa1a" containerName="istio-proxy" Oct 29 23:18:59.269064 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.268979 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e794e7ae-93a6-4789-9542-217137d0aa1a" containerName="istio-proxy" Oct 29 23:18:59.269064 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.269030 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7ea22168-c52a-43f3-a4b6-4cdd87cffdae" containerName="istio-proxy" Oct 29 23:18:59.269064 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.269038 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea22168-c52a-43f3-a4b6-4cdd87cffdae" containerName="istio-proxy" Oct 29 23:18:59.269226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.269205 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="503f6577-e8f6-474e-8b57-98e30edaa785" containerName="istio-proxy" Oct 29 23:18:59.269226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.269220 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e794e7ae-93a6-4789-9542-217137d0aa1a" containerName="istio-proxy" Oct 29 23:18:59.269303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.269240 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea22168-c52a-43f3-a4b6-4cdd87cffdae" containerName="istio-proxy" Oct 29 23:18:59.269303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.269253 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a9c05b-ea5f-4f38-8043-a3ed71754e1a" containerName="istio-proxy" Oct 29 23:18:59.273530 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.273510 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs" Oct 29 23:18:59.291011 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.290984 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs"] Oct 29 23:18:59.300281 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.300257 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-26b322-dockercfg-psvt2\"" Oct 29 23:18:59.300440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.300337 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-26b322\"" Oct 29 23:18:59.413211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.410484 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdkzj\" (UniqueName: \"kubernetes.io/projected/139df2e5-deb8-44b0-8ed5-fb76650beb06-kube-api-access-vdkzj\") pod \"payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs\" (UID: \"139df2e5-deb8-44b0-8ed5-fb76650beb06\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs" Oct 29 23:18:59.413211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.410654 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/139df2e5-deb8-44b0-8ed5-fb76650beb06-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs\" (UID: \"139df2e5-deb8-44b0-8ed5-fb76650beb06\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs" Oct 29 23:18:59.605069 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.593511 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdkzj\" (UniqueName: \"kubernetes.io/projected/139df2e5-deb8-44b0-8ed5-fb76650beb06-kube-api-access-vdkzj\") pod \"payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs\" (UID: \"139df2e5-deb8-44b0-8ed5-fb76650beb06\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs" Oct 29 23:18:59.609913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.605415 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/139df2e5-deb8-44b0-8ed5-fb76650beb06-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs\" (UID: \"139df2e5-deb8-44b0-8ed5-fb76650beb06\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs" Oct 29 23:18:59.639216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.628945 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/139df2e5-deb8-44b0-8ed5-fb76650beb06-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs\" (UID: \"139df2e5-deb8-44b0-8ed5-fb76650beb06\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs" Oct 29 23:18:59.719892 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.719856 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdkzj\" (UniqueName: \"kubernetes.io/projected/139df2e5-deb8-44b0-8ed5-fb76650beb06-kube-api-access-vdkzj\") pod \"payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs\" (UID: \"139df2e5-deb8-44b0-8ed5-fb76650beb06\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs" Oct 29 23:18:59.735216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.727130 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx"] Oct 29 23:18:59.739421 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.739394 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx"] Oct 29 23:18:59.739765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.739701 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx" Oct 29 23:18:59.820126 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.820096 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec6c14d-ff01-457a-abaf-8eab25418539-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx\" (UID: \"1ec6c14d-ff01-457a-abaf-8eab25418539\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx" Oct 29 23:18:59.820327 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.820225 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cbcx\" (UniqueName: \"kubernetes.io/projected/1ec6c14d-ff01-457a-abaf-8eab25418539-kube-api-access-9cbcx\") pod \"payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx\" (UID: \"1ec6c14d-ff01-457a-abaf-8eab25418539\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx" Oct 29 23:18:59.888742 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.888661 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs" Oct 29 23:18:59.921919 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.921880 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec6c14d-ff01-457a-abaf-8eab25418539-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx\" (UID: \"1ec6c14d-ff01-457a-abaf-8eab25418539\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx" Oct 29 23:18:59.922083 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.921994 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cbcx\" (UniqueName: \"kubernetes.io/projected/1ec6c14d-ff01-457a-abaf-8eab25418539-kube-api-access-9cbcx\") pod \"payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx\" (UID: \"1ec6c14d-ff01-457a-abaf-8eab25418539\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx" Oct 29 23:18:59.922692 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.922671 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec6c14d-ff01-457a-abaf-8eab25418539-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx\" (UID: \"1ec6c14d-ff01-457a-abaf-8eab25418539\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx" Oct 29 23:18:59.947963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:18:59.947936 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cbcx\" (UniqueName: \"kubernetes.io/projected/1ec6c14d-ff01-457a-abaf-8eab25418539-kube-api-access-9cbcx\") pod \"payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx\" (UID: \"1ec6c14d-ff01-457a-abaf-8eab25418539\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx" Oct 29 23:19:00.071795 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:00.071744 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx" Oct 29 23:19:00.280292 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:00.280265 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs"] Oct 29 23:19:00.281042 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:19:00.281016 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139df2e5_deb8_44b0_8ed5_fb76650beb06.slice/crio-552799aa90d31e505867a8aa169cca4a5cc9d694e8b410b67d1b343f9670abe2 WatchSource:0}: Error finding container 552799aa90d31e505867a8aa169cca4a5cc9d694e8b410b67d1b343f9670abe2: Status 404 returned error can't find the container with id 552799aa90d31e505867a8aa169cca4a5cc9d694e8b410b67d1b343f9670abe2 Oct 29 23:19:00.494625 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:19:00.494554 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec6c14d_ff01_457a_abaf_8eab25418539.slice/crio-22e9b7995424d334ab5b62b4dfa9c52a7f68841c3cef5d64f19ecd543a20bc54 WatchSource:0}: Error finding container 22e9b7995424d334ab5b62b4dfa9c52a7f68841c3cef5d64f19ecd543a20bc54: Status 404 returned error can't find the container with id 22e9b7995424d334ab5b62b4dfa9c52a7f68841c3cef5d64f19ecd543a20bc54 Oct 29 23:19:00.494625 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:00.494585 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx"] Oct 29 23:19:00.980245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:00.980161 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx" event={"ID":"1ec6c14d-ff01-457a-abaf-8eab25418539","Type":"ContainerStarted","Data":"cb43dc6f5b6e89126f731f28e5f7e4c3317114aa9e8528cf397854c54cfce3c4"} Oct 29 23:19:00.980427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:00.980256 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx" event={"ID":"1ec6c14d-ff01-457a-abaf-8eab25418539","Type":"ContainerStarted","Data":"22e9b7995424d334ab5b62b4dfa9c52a7f68841c3cef5d64f19ecd543a20bc54"} Oct 29 23:19:00.980427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:00.980287 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx" Oct 29 23:19:00.981933 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:00.981908 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs" event={"ID":"139df2e5-deb8-44b0-8ed5-fb76650beb06","Type":"ContainerStarted","Data":"820fbfdbeed78f266900f1b1796951e73c390513a8e1338d5618eefca1078777"} Oct 29 23:19:00.982073 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:00.981947 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs" event={"ID":"139df2e5-deb8-44b0-8ed5-fb76650beb06","Type":"ContainerStarted","Data":"552799aa90d31e505867a8aa169cca4a5cc9d694e8b410b67d1b343f9670abe2"} Oct 29 23:19:00.982073 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:00.982053 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs" Oct 29 23:19:01.026429 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:01.026375 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx" podStartSLOduration=2.026358013 podStartE2EDuration="2.026358013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:01.024118027 +0000 UTC m=+2187.675616558" watchObservedRunningTime="2026-10-29 23:19:01.026358013 +0000 UTC m=+2187.677856544" Oct 29 23:19:01.068076 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:01.068023 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs" podStartSLOduration=2.068006106 podStartE2EDuration="2.068006106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:01.067912229 +0000 UTC m=+2187.719410760" watchObservedRunningTime="2026-10-29 23:19:01.068006106 +0000 UTC m=+2187.719504637" Oct 29 23:19:03.915003 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:19:03.914964 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:19:06.641430 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:06.641384 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:19:09.515209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:09.514644 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" podUID="0014c879-9ec4-4197-984a-f10a20b6510e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.226:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:19:09.562964 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:09.561418 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" podUID="0014c879-9ec4-4197-984a-f10a20b6510e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.226:8000/health\": context deadline exceeded" Oct 29 23:19:12.026428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:12.026394 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs" Oct 29 23:19:12.032416 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:12.032386 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx" Oct 29 23:19:17.075466 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:17.075365 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7896-fc3f6c"] Oct 29 23:19:17.081443 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:17.081419 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-fc3f6c" Oct 29 23:19:17.107690 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:17.107662 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xdwj4\"" Oct 29 23:19:17.128239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:17.128209 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-fc3f6c"] Oct 29 23:19:17.205479 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:17.205433 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mscrt\" (UniqueName: \"kubernetes.io/projected/9bb7f656-b5ef-46f0-8091-f7a328dd612a-kube-api-access-mscrt\") pod \"test-curl-7896-fc3f6c\" (UID: \"9bb7f656-b5ef-46f0-8091-f7a328dd612a\") " pod="opendatahub/test-curl-7896-fc3f6c" Oct 29 23:19:17.307230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:17.307164 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mscrt\" (UniqueName: \"kubernetes.io/projected/9bb7f656-b5ef-46f0-8091-f7a328dd612a-kube-api-access-mscrt\") pod \"test-curl-7896-fc3f6c\" (UID: \"9bb7f656-b5ef-46f0-8091-f7a328dd612a\") " pod="opendatahub/test-curl-7896-fc3f6c" Oct 29 23:19:17.337069 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:17.336950 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mscrt\" (UniqueName: \"kubernetes.io/projected/9bb7f656-b5ef-46f0-8091-f7a328dd612a-kube-api-access-mscrt\") pod \"test-curl-7896-fc3f6c\" (UID: \"9bb7f656-b5ef-46f0-8091-f7a328dd612a\") " pod="opendatahub/test-curl-7896-fc3f6c" Oct 29 23:19:17.397444 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:17.397405 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-fc3f6c" Oct 29 23:19:17.742247 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:19:17.739581 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bb7f656_b5ef_46f0_8091_f7a328dd612a.slice/crio-905b9cd78ca16a9a837a49df0b30b58110555d199a11d31dddd59808eda9036d WatchSource:0}: Error finding container 905b9cd78ca16a9a837a49df0b30b58110555d199a11d31dddd59808eda9036d: Status 404 returned error can't find the container with id 905b9cd78ca16a9a837a49df0b30b58110555d199a11d31dddd59808eda9036d Oct 29 23:19:17.742247 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:17.739696 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-fc3f6c"] Oct 29 23:19:18.525360 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:18.525314 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:19:18.525861 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:18.525380 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" podUID="f255ae1a-010b-4f50-9414-5b09b507c085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:19:18.561399 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:18.561359 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-fc3f6c" event={"ID":"9bb7f656-b5ef-46f0-8091-f7a328dd612a","Type":"ContainerStarted","Data":"6349c6a23797c1f153944f671fdbb096383e554738d16805278585d5f618ea13"} Oct 29 23:19:18.561644 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:18.561624 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-fc3f6c" event={"ID":"9bb7f656-b5ef-46f0-8091-f7a328dd612a","Type":"ContainerStarted","Data":"905b9cd78ca16a9a837a49df0b30b58110555d199a11d31dddd59808eda9036d"} Oct 29 23:19:19.887815 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:19.887754 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7896-fc3f6c" podStartSLOduration=2.887734948 podStartE2EDuration="2.887734948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:18.641498473 +0000 UTC m=+2205.292997003" watchObservedRunningTime="2026-10-29 23:19:19.887734948 +0000 UTC m=+2206.539233482" Oct 29 23:19:19.888587 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:19.888557 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7896-fc3f6c"] Oct 29 23:19:19.893950 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:19.893921 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7896-fc3f6c"] Oct 29 23:19:20.054123 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.054089 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7896-945216"] Oct 29 23:19:20.055157 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.055127 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9bb7f656-b5ef-46f0-8091-f7a328dd612a" containerName="test-curl-7896-fc3f6c" Oct 29 23:19:20.055157 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.055154 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb7f656-b5ef-46f0-8091-f7a328dd612a" containerName="test-curl-7896-fc3f6c" Oct 29 23:19:20.055388 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.055373 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb7f656-b5ef-46f0-8091-f7a328dd612a" containerName="test-curl-7896-fc3f6c" Oct 29 23:19:20.072400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.072377 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-945216" Oct 29 23:19:20.078381 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.078353 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-945216"] Oct 29 23:19:20.211115 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.211074 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wxkq\" (UniqueName: \"kubernetes.io/projected/3dec173b-55e2-4f74-85fc-33a885f3d4e5-kube-api-access-6wxkq\") pod \"test-curl-7896-945216\" (UID: \"3dec173b-55e2-4f74-85fc-33a885f3d4e5\") " pod="opendatahub/test-curl-7896-945216" Oct 29 23:19:20.312905 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.312867 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wxkq\" (UniqueName: \"kubernetes.io/projected/3dec173b-55e2-4f74-85fc-33a885f3d4e5-kube-api-access-6wxkq\") pod \"test-curl-7896-945216\" (UID: \"3dec173b-55e2-4f74-85fc-33a885f3d4e5\") " pod="opendatahub/test-curl-7896-945216" Oct 29 23:19:20.337676 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.337645 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wxkq\" (UniqueName: \"kubernetes.io/projected/3dec173b-55e2-4f74-85fc-33a885f3d4e5-kube-api-access-6wxkq\") pod \"test-curl-7896-945216\" (UID: \"3dec173b-55e2-4f74-85fc-33a885f3d4e5\") " pod="opendatahub/test-curl-7896-945216" Oct 29 23:19:20.399181 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.399109 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-945216" Oct 29 23:19:20.647345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.647308 5057 generic.go:356] "Generic (PLEG): container finished" podID="f255ae1a-010b-4f50-9414-5b09b507c085" containerID="12f9dddeba1de36d011970ce9fc27ed207ddb84b6c30fd3e37d49eafd3881b41" exitCode=0 Oct 29 23:19:20.647535 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.647384 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" event={"ID":"f255ae1a-010b-4f50-9414-5b09b507c085","Type":"ContainerDied","Data":"12f9dddeba1de36d011970ce9fc27ed207ddb84b6c30fd3e37d49eafd3881b41"} Oct 29 23:19:20.647960 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.647935 5057 scope.go:122] "RemoveContainer" containerID="12f9dddeba1de36d011970ce9fc27ed207ddb84b6c30fd3e37d49eafd3881b41" Oct 29 23:19:20.649464 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.649440 5057 generic.go:356] "Generic (PLEG): container finished" podID="fd6e00d3-c546-4fec-82fd-98c420fcca2d" containerID="59396594e1193249bbbe8a320ddba6f2d133c40082212650848d4e4589440d1d" exitCode=0 Oct 29 23:19:20.649588 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.649515 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" event={"ID":"fd6e00d3-c546-4fec-82fd-98c420fcca2d","Type":"ContainerDied","Data":"59396594e1193249bbbe8a320ddba6f2d133c40082212650848d4e4589440d1d"} Oct 29 23:19:20.649664 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.649644 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7896-fc3f6c" podUID="9bb7f656-b5ef-46f0-8091-f7a328dd612a" containerName="test-curl-7896-fc3f6c" containerID="cri-o://6349c6a23797c1f153944f671fdbb096383e554738d16805278585d5f618ea13" gracePeriod=2 Oct 29 23:19:20.650015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.649998 5057 scope.go:122] "RemoveContainer" containerID="59396594e1193249bbbe8a320ddba6f2d133c40082212650848d4e4589440d1d" Oct 29 23:19:20.650258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.650221 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:19:20.650344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.650314 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:19:20.650412 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.650378 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:19:20.652529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.652494 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:19:20.652636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.652585 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:19:20.652684 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.652633 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:19:20.710728 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:20.710701 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-945216"] Oct 29 23:19:20.713268 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:19:20.713244 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3dec173b_55e2_4f74_85fc_33a885f3d4e5.slice/crio-7de1e6267b91494074ed4dcca3193300fb7aa81d10b6ebf917843b9a534f90e1 WatchSource:0}: Error finding container 7de1e6267b91494074ed4dcca3193300fb7aa81d10b6ebf917843b9a534f90e1: Status 404 returned error can't find the container with id 7de1e6267b91494074ed4dcca3193300fb7aa81d10b6ebf917843b9a534f90e1 Oct 29 23:19:20.725583 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:19:20.725415 5057 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7896-fc3f6c\" is forbidden: User \"system:node:ip-10-0-135-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-80.ec2.internal' and this object" podUID="9bb7f656-b5ef-46f0-8091-f7a328dd612a" pod="opendatahub/test-curl-7896-fc3f6c" Oct 29 23:19:21.715222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:21.708292 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-945216" event={"ID":"3dec173b-55e2-4f74-85fc-33a885f3d4e5","Type":"ContainerStarted","Data":"09a3b27abc92b4a9f0da9e31c815aa6c279c1e3b2f6c31b8a4a85333d5ec9bd3"} Oct 29 23:19:21.715222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:21.708347 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-945216" event={"ID":"3dec173b-55e2-4f74-85fc-33a885f3d4e5","Type":"ContainerStarted","Data":"7de1e6267b91494074ed4dcca3193300fb7aa81d10b6ebf917843b9a534f90e1"} Oct 29 23:19:21.747873 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:21.747834 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" event={"ID":"f255ae1a-010b-4f50-9414-5b09b507c085","Type":"ContainerStarted","Data":"4942e4c8f4e388ec45f65eb53e1f8a53221dbc5e62fd4660e2abde9e34e227aa"} Oct 29 23:19:21.771660 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:21.771618 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" event={"ID":"fd6e00d3-c546-4fec-82fd-98c420fcca2d","Type":"ContainerStarted","Data":"74d3c6f4e38639e4b3e9dc8b3b013a5bcb4ff5d233bfe8f351a3cf62ca81a823"} Oct 29 23:19:21.782209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:21.780853 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7896-945216" podStartSLOduration=1.780834078 podStartE2EDuration="1.780834078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:21.773008622 +0000 UTC m=+2208.424507153" watchObservedRunningTime="2026-10-29 23:19:21.780834078 +0000 UTC m=+2208.432332610" Oct 29 23:19:22.292701 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:22.292656 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:19:22.294025 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:22.294001 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 29 23:19:22.294118 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:22.294049 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" podUID="fd6e00d3-c546-4fec-82fd-98c420fcca2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 29 23:19:22.701589 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:22.701561 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:19:22.702093 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:22.702064 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 29 23:19:22.702244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:22.702120 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" podUID="f255ae1a-010b-4f50-9414-5b09b507c085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 29 23:19:22.796065 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:22.796032 5057 generic.go:356] "Generic (PLEG): container finished" podID="9bb7f656-b5ef-46f0-8091-f7a328dd612a" containerID="6349c6a23797c1f153944f671fdbb096383e554738d16805278585d5f618ea13" exitCode=137 Oct 29 23:19:22.906605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:22.899567 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7896-945216"] Oct 29 23:19:22.996315 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:22.918737 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7896-945216"] Oct 29 23:19:23.005147 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.005111 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7896-29b51d"] Oct 29 23:19:23.007165 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.007132 5057 cpu_manager.go:414] "containerMap: removing container" podUID="3dec173b-55e2-4f74-85fc-33a885f3d4e5" containerName="test-curl-7896-945216" Oct 29 23:19:23.007165 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.007161 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dec173b-55e2-4f74-85fc-33a885f3d4e5" containerName="test-curl-7896-945216" Oct 29 23:19:23.007423 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.007408 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dec173b-55e2-4f74-85fc-33a885f3d4e5" containerName="test-curl-7896-945216" Oct 29 23:19:23.010740 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.010715 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-29b51d" Oct 29 23:19:23.097280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.097010 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-29b51d"] Oct 29 23:19:23.117205 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.116765 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9zwx\" (UniqueName: \"kubernetes.io/projected/b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2-kube-api-access-t9zwx\") pod \"test-curl-7896-29b51d\" (UID: \"b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2\") " pod="opendatahub/test-curl-7896-29b51d" Oct 29 23:19:23.218916 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.218876 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9zwx\" (UniqueName: \"kubernetes.io/projected/b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2-kube-api-access-t9zwx\") pod \"test-curl-7896-29b51d\" (UID: \"b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2\") " pod="opendatahub/test-curl-7896-29b51d" Oct 29 23:19:23.295377 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.295128 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 29 23:19:23.295742 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.295712 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" podUID="fd6e00d3-c546-4fec-82fd-98c420fcca2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 29 23:19:23.300900 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.300704 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-fc3f6c" Oct 29 23:19:23.302687 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.302630 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9zwx\" (UniqueName: \"kubernetes.io/projected/b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2-kube-api-access-t9zwx\") pod \"test-curl-7896-29b51d\" (UID: \"b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2\") " pod="opendatahub/test-curl-7896-29b51d" Oct 29 23:19:23.348211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.321134 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb7f656-b5ef-46f0-8091-f7a328dd612a-kube-api-access-mscrt\" (UniqueName: \"kubernetes.io/projected/9bb7f656-b5ef-46f0-8091-f7a328dd612a-kube-api-access-mscrt\") pod \"9bb7f656-b5ef-46f0-8091-f7a328dd612a\" (UID: \"9bb7f656-b5ef-46f0-8091-f7a328dd612a\") " Oct 29 23:19:23.388065 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.387930 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb7f656-b5ef-46f0-8091-f7a328dd612a-kube-api-access-mscrt" pod "9bb7f656-b5ef-46f0-8091-f7a328dd612a" (UID: "9bb7f656-b5ef-46f0-8091-f7a328dd612a"). InnerVolumeSpecName "kube-api-access-mscrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:23.419235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.419206 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-29b51d" Oct 29 23:19:23.459216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.445703 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mscrt\" (UniqueName: \"kubernetes.io/projected/9bb7f656-b5ef-46f0-8091-f7a328dd612a-kube-api-access-mscrt\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:19:23.496928 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.496904 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb7f656-b5ef-46f0-8091-f7a328dd612a" path="/var/lib/kubelet/pods/9bb7f656-b5ef-46f0-8091-f7a328dd612a/volumes" Oct 29 23:19:23.703374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.703342 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 29 23:19:23.703670 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.703642 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" podUID="f255ae1a-010b-4f50-9414-5b09b507c085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 29 23:19:23.812829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.812807 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-fc3f6c" Oct 29 23:19:23.813472 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.812861 5057 scope.go:122] "RemoveContainer" containerID="6349c6a23797c1f153944f671fdbb096383e554738d16805278585d5f618ea13" Oct 29 23:19:23.813472 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.813099 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7896-945216" podUID="3dec173b-55e2-4f74-85fc-33a885f3d4e5" containerName="test-curl-7896-945216" containerID="cri-o://09a3b27abc92b4a9f0da9e31c815aa6c279c1e3b2f6c31b8a4a85333d5ec9bd3" gracePeriod=2 Oct 29 23:19:23.992494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:23.992449 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-29b51d"] Oct 29 23:19:23.994079 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:19:23.994050 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3ce3ab0_ad4f_4db3_99c4_c0fe1b2af9d2.slice/crio-0c1b6004cce6db51a466baee7f2b02525b489773fefb47ee298a17ed1afbab8b WatchSource:0}: Error finding container 0c1b6004cce6db51a466baee7f2b02525b489773fefb47ee298a17ed1afbab8b: Status 404 returned error can't find the container with id 0c1b6004cce6db51a466baee7f2b02525b489773fefb47ee298a17ed1afbab8b Oct 29 23:19:24.293936 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:24.293893 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 29 23:19:24.294140 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:24.293945 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" podUID="fd6e00d3-c546-4fec-82fd-98c420fcca2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 29 23:19:24.701225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:24.701172 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 29 23:19:24.701408 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:24.701254 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" podUID="f255ae1a-010b-4f50-9414-5b09b507c085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 29 23:19:24.894899 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:24.894851 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-29b51d" event={"ID":"b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2","Type":"ContainerStarted","Data":"917c1f76f52b8de2d426476cfe9434ec71d66e8681b3d60368a7c583d76af64f"} Oct 29 23:19:24.894899 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:24.894902 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-29b51d" event={"ID":"b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2","Type":"ContainerStarted","Data":"0c1b6004cce6db51a466baee7f2b02525b489773fefb47ee298a17ed1afbab8b"} Oct 29 23:19:24.977815 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:24.977706 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7896-29b51d" podStartSLOduration=2.977685353 podStartE2EDuration="2.977685353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:24.955622528 +0000 UTC m=+2211.607121061" watchObservedRunningTime="2026-10-29 23:19:24.977685353 +0000 UTC m=+2211.629183884" Oct 29 23:19:25.293607 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:25.293056 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 29 23:19:25.293607 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:25.293134 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" podUID="fd6e00d3-c546-4fec-82fd-98c420fcca2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 29 23:19:25.702223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:25.701816 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 29 23:19:25.702223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:25.701877 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" podUID="f255ae1a-010b-4f50-9414-5b09b507c085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 29 23:19:26.101871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.101723 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7896-29b51d"] Oct 29 23:19:26.109769 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.109739 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7896-29b51d"] Oct 29 23:19:26.292339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.292291 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7896-c01760"] Oct 29 23:19:26.293596 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.293561 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2" containerName="test-curl-7896-29b51d" Oct 29 23:19:26.293732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.293642 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2" containerName="test-curl-7896-29b51d" Oct 29 23:19:26.293949 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.293932 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2" containerName="test-curl-7896-29b51d" Oct 29 23:19:26.295577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.295546 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 29 23:19:26.295688 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.295600 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" podUID="fd6e00d3-c546-4fec-82fd-98c420fcca2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 29 23:19:26.297015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.296994 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-c01760"] Oct 29 23:19:26.297122 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.297070 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-c01760" Oct 29 23:19:26.371656 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.371557 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2njsk\" (UniqueName: \"kubernetes.io/projected/c53331ab-bf10-432b-b265-e3e18ab7bf78-kube-api-access-2njsk\") pod \"test-curl-7896-c01760\" (UID: \"c53331ab-bf10-432b-b265-e3e18ab7bf78\") " pod="opendatahub/test-curl-7896-c01760" Oct 29 23:19:26.486532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.486489 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2njsk\" (UniqueName: \"kubernetes.io/projected/c53331ab-bf10-432b-b265-e3e18ab7bf78-kube-api-access-2njsk\") pod \"test-curl-7896-c01760\" (UID: \"c53331ab-bf10-432b-b265-e3e18ab7bf78\") " pod="opendatahub/test-curl-7896-c01760" Oct 29 23:19:26.511730 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.511700 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2njsk\" (UniqueName: \"kubernetes.io/projected/c53331ab-bf10-432b-b265-e3e18ab7bf78-kube-api-access-2njsk\") pod \"test-curl-7896-c01760\" (UID: \"c53331ab-bf10-432b-b265-e3e18ab7bf78\") " pod="opendatahub/test-curl-7896-c01760" Oct 29 23:19:26.610074 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.609879 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-c01760" Oct 29 23:19:26.701949 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.701462 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 29 23:19:26.701949 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.701526 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" podUID="f255ae1a-010b-4f50-9414-5b09b507c085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 29 23:19:26.915021 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.905672 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-945216" Oct 29 23:19:26.994279 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.993892 5057 generic.go:356] "Generic (PLEG): container finished" podID="3dec173b-55e2-4f74-85fc-33a885f3d4e5" containerID="09a3b27abc92b4a9f0da9e31c815aa6c279c1e3b2f6c31b8a4a85333d5ec9bd3" exitCode=137 Oct 29 23:19:26.994461 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.994275 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7896-29b51d" podUID="b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2" containerName="test-curl-7896-29b51d" containerID="cri-o://917c1f76f52b8de2d426476cfe9434ec71d66e8681b3d60368a7c583d76af64f" gracePeriod=2 Oct 29 23:19:26.994461 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.994433 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-945216" Oct 29 23:19:26.995229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:26.994676 5057 scope.go:122] "RemoveContainer" containerID="09a3b27abc92b4a9f0da9e31c815aa6c279c1e3b2f6c31b8a4a85333d5ec9bd3" Oct 29 23:19:27.009048 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:27.009020 5057 scope.go:122] "RemoveContainer" containerID="09a3b27abc92b4a9f0da9e31c815aa6c279c1e3b2f6c31b8a4a85333d5ec9bd3" Oct 29 23:19:27.009459 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:19:27.009432 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09a3b27abc92b4a9f0da9e31c815aa6c279c1e3b2f6c31b8a4a85333d5ec9bd3\": container with ID starting with 09a3b27abc92b4a9f0da9e31c815aa6c279c1e3b2f6c31b8a4a85333d5ec9bd3 not found: ID does not exist" containerID="09a3b27abc92b4a9f0da9e31c815aa6c279c1e3b2f6c31b8a4a85333d5ec9bd3" Oct 29 23:19:27.009507 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:27.009472 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09a3b27abc92b4a9f0da9e31c815aa6c279c1e3b2f6c31b8a4a85333d5ec9bd3"} err="failed to get container status \"09a3b27abc92b4a9f0da9e31c815aa6c279c1e3b2f6c31b8a4a85333d5ec9bd3\": rpc error: code = NotFound desc = could not find container \"09a3b27abc92b4a9f0da9e31c815aa6c279c1e3b2f6c31b8a4a85333d5ec9bd3\": container with ID starting with 09a3b27abc92b4a9f0da9e31c815aa6c279c1e3b2f6c31b8a4a85333d5ec9bd3 not found: ID does not exist" Oct 29 23:19:27.010450 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:19:27.010424 5057 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7896-29b51d\" is forbidden: User \"system:node:ip-10-0-135-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-80.ec2.internal' and this object" podUID="b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2" pod="opendatahub/test-curl-7896-29b51d" Oct 29 23:19:27.019323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:27.019286 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dec173b-55e2-4f74-85fc-33a885f3d4e5-kube-api-access-6wxkq\" (UniqueName: \"kubernetes.io/projected/3dec173b-55e2-4f74-85fc-33a885f3d4e5-kube-api-access-6wxkq\") pod \"3dec173b-55e2-4f74-85fc-33a885f3d4e5\" (UID: \"3dec173b-55e2-4f74-85fc-33a885f3d4e5\") " Oct 29 23:19:27.032209 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:19:27.026414 5057 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7896-29b51d\" is forbidden: User \"system:node:ip-10-0-135-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-80.ec2.internal' and this object" podUID="b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2" pod="opendatahub/test-curl-7896-29b51d" Oct 29 23:19:27.092334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:27.085702 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dec173b-55e2-4f74-85fc-33a885f3d4e5-kube-api-access-6wxkq" pod "3dec173b-55e2-4f74-85fc-33a885f3d4e5" (UID: "3dec173b-55e2-4f74-85fc-33a885f3d4e5"). InnerVolumeSpecName "kube-api-access-6wxkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:27.095037 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:27.095013 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-c01760"] Oct 29 23:19:27.123642 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:27.123078 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wxkq\" (UniqueName: \"kubernetes.io/projected/3dec173b-55e2-4f74-85fc-33a885f3d4e5-kube-api-access-6wxkq\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:19:27.293596 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:27.293512 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 29 23:19:27.293596 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:27.293572 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" podUID="fd6e00d3-c546-4fec-82fd-98c420fcca2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 29 23:19:27.318902 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:19:27.318845 5057 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7896-945216\" is forbidden: User \"system:node:ip-10-0-135-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-80.ec2.internal' and this object" podUID="3dec173b-55e2-4f74-85fc-33a885f3d4e5" pod="opendatahub/test-curl-7896-945216" Oct 29 23:19:27.334116 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:19:27.334071 5057 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7896-29b51d\" is forbidden: User \"system:node:ip-10-0-135-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-80.ec2.internal' and this object" podUID="b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2" pod="opendatahub/test-curl-7896-29b51d" Oct 29 23:19:27.483218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:27.483160 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dec173b-55e2-4f74-85fc-33a885f3d4e5" path="/var/lib/kubelet/pods/3dec173b-55e2-4f74-85fc-33a885f3d4e5/volumes" Oct 29 23:19:27.706014 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:27.705987 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:19:27.706495 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:27.706475 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:19:27.707156 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:27.707136 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:19:28.023964 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:28.023874 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-c01760" event={"ID":"c53331ab-bf10-432b-b265-e3e18ab7bf78","Type":"ContainerStarted","Data":"c6f7efccdc33306d4092875d532c4017a7f1fa6a3d813d210c8c4857a21e6a8c"} Oct 29 23:19:28.023964 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:28.023917 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-c01760" event={"ID":"c53331ab-bf10-432b-b265-e3e18ab7bf78","Type":"ContainerStarted","Data":"c4f646d40e3ef8536a56f59f0f36bac6da7a039691fd804e68ea41e8ab679a62"} Oct 29 23:19:28.069485 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:28.069433 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7896-c01760" podStartSLOduration=2.069418956 podStartE2EDuration="2.069418956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:28.066769667 +0000 UTC m=+2214.718268198" watchObservedRunningTime="2026-10-29 23:19:28.069418956 +0000 UTC m=+2214.720917486" Oct 29 23:19:28.298770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:28.298699 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:19:28.299244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:28.299224 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:19:28.300081 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:28.300060 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:19:28.991630 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:28.991592 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7896-c01760"] Oct 29 23:19:28.996949 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:28.996921 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7896-c01760"] Oct 29 23:19:29.042538 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:29.042497 5057 generic.go:356] "Generic (PLEG): container finished" podID="b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2" containerID="917c1f76f52b8de2d426476cfe9434ec71d66e8681b3d60368a7c583d76af64f" exitCode=137 Oct 29 23:19:29.235583 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:29.235552 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-29b51d" Oct 29 23:19:29.375230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:29.375123 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2-kube-api-access-t9zwx\" (UniqueName: \"kubernetes.io/projected/b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2-kube-api-access-t9zwx\") pod \"b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2\" (UID: \"b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2\") " Oct 29 23:19:29.377339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:29.377303 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2-kube-api-access-t9zwx" pod "b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2" (UID: "b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2"). InnerVolumeSpecName "kube-api-access-t9zwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:29.477337 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:29.477291 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9zwx\" (UniqueName: \"kubernetes.io/projected/b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2-kube-api-access-t9zwx\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:19:29.484082 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:29.484049 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2" path="/var/lib/kubelet/pods/b3ce3ab0-ad4f-4db3-99c4-c0fe1b2af9d2/volumes" Oct 29 23:19:30.053505 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:30.053459 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-29b51d" Oct 29 23:19:30.053726 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:30.053532 5057 scope.go:122] "RemoveContainer" containerID="917c1f76f52b8de2d426476cfe9434ec71d66e8681b3d60368a7c583d76af64f" Oct 29 23:19:30.053726 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:30.053636 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7896-c01760" podUID="c53331ab-bf10-432b-b265-e3e18ab7bf78" containerName="test-curl-7896-c01760" containerID="cri-o://c6f7efccdc33306d4092875d532c4017a7f1fa6a3d813d210c8c4857a21e6a8c" gracePeriod=2 Oct 29 23:19:31.061283 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.060083 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk"] Oct 29 23:19:31.061810 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.061450 5057 cpu_manager.go:414] "containerMap: removing container" podUID="c53331ab-bf10-432b-b265-e3e18ab7bf78" containerName="test-curl-7896-c01760" Oct 29 23:19:31.061810 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.061470 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53331ab-bf10-432b-b265-e3e18ab7bf78" containerName="test-curl-7896-c01760" Oct 29 23:19:31.061810 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.061756 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53331ab-bf10-432b-b265-e3e18ab7bf78" containerName="test-curl-7896-c01760" Oct 29 23:19:31.071508 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.071477 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.096788 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.096755 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk"] Oct 29 23:19:31.104809 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.104779 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-e1769a-openshift-default-dockercfg-tdhlt\"" Oct 29 23:19:31.213167 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.213053 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-podinfo\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.213512 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.213483 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-data\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.213846 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.213812 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-workload-certs\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.213957 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.213862 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-envoy\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.213957 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.213896 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-token\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.214083 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.213934 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-credential-socket\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.214083 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.214014 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c575af2-2a8a-4e83-adcc-7d6578c53235-istiod-ca-cert\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.214083 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.214047 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-workload-socket\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.216346 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.214202 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9wmn\" (UniqueName: \"kubernetes.io/projected/5c575af2-2a8a-4e83-adcc-7d6578c53235-kube-api-access-d9wmn\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.328321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.327443 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-workload-certs\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.328321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.327510 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-envoy\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.328321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.327545 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-token\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.328321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.327580 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-credential-socket\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.328321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.327620 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c575af2-2a8a-4e83-adcc-7d6578c53235-istiod-ca-cert\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.328321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.327653 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-workload-socket\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.328321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.327758 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9wmn\" (UniqueName: \"kubernetes.io/projected/5c575af2-2a8a-4e83-adcc-7d6578c53235-kube-api-access-d9wmn\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.328321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.327828 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-podinfo\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.328321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.327897 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-data\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.328930 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.328658 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-data\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.329000 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.328957 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-workload-certs\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.332210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.329833 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c575af2-2a8a-4e83-adcc-7d6578c53235-istiod-ca-cert\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.332210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.330398 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-credential-socket\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.332210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.331098 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-workload-socket\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.394837 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.379680 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-token\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.394837 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.380820 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-podinfo\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.394837 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.386526 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-envoy\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.394837 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.387299 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9wmn\" (UniqueName: \"kubernetes.io/projected/5c575af2-2a8a-4e83-adcc-7d6578c53235-kube-api-access-d9wmn\") pod \"e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.394837 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.391454 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:31.899353 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:19:31.899327 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c575af2_2a8a_4e83_adcc_7d6578c53235.slice/crio-5b88c04b59a186a3fa38f82bb6ff617bf0df5f1ddb7e4094324fde5fc1280756 WatchSource:0}: Error finding container 5b88c04b59a186a3fa38f82bb6ff617bf0df5f1ddb7e4094324fde5fc1280756: Status 404 returned error can't find the container with id 5b88c04b59a186a3fa38f82bb6ff617bf0df5f1ddb7e4094324fde5fc1280756 Oct 29 23:19:31.900025 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.900000 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk"] Oct 29 23:19:31.902325 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.902289 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:19:31.902416 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.902360 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:19:31.902416 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:31.902391 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:19:32.083127 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:32.083088 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" event={"ID":"5c575af2-2a8a-4e83-adcc-7d6578c53235","Type":"ContainerStarted","Data":"c0db97b4b8cac18eb5f7eca8e2f9046c924cd55e37f6ee5d7299ba0c009eb3d9"} Oct 29 23:19:32.083621 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:32.083156 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" event={"ID":"5c575af2-2a8a-4e83-adcc-7d6578c53235","Type":"ContainerStarted","Data":"5b88c04b59a186a3fa38f82bb6ff617bf0df5f1ddb7e4094324fde5fc1280756"} Oct 29 23:19:32.142304 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:32.139017 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" podStartSLOduration=1.1389972959999999 podStartE2EDuration="1.138997296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:32.134017799 +0000 UTC m=+2218.785516328" watchObservedRunningTime="2026-10-29 23:19:32.138997296 +0000 UTC m=+2218.790495826" Oct 29 23:19:32.392547 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:32.392510 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:32.419312 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:32.419286 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-c01760" Oct 29 23:19:32.484869 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:32.484790 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53331ab-bf10-432b-b265-e3e18ab7bf78-kube-api-access-2njsk\" (UniqueName: \"kubernetes.io/projected/c53331ab-bf10-432b-b265-e3e18ab7bf78-kube-api-access-2njsk\") pod \"c53331ab-bf10-432b-b265-e3e18ab7bf78\" (UID: \"c53331ab-bf10-432b-b265-e3e18ab7bf78\") " Oct 29 23:19:32.487060 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:32.487028 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53331ab-bf10-432b-b265-e3e18ab7bf78-kube-api-access-2njsk" pod "c53331ab-bf10-432b-b265-e3e18ab7bf78" (UID: "c53331ab-bf10-432b-b265-e3e18ab7bf78"). InnerVolumeSpecName "kube-api-access-2njsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:32.587209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:32.587162 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2njsk\" (UniqueName: \"kubernetes.io/projected/c53331ab-bf10-432b-b265-e3e18ab7bf78-kube-api-access-2njsk\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:19:33.093139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:33.093106 5057 generic.go:356] "Generic (PLEG): container finished" podID="c53331ab-bf10-432b-b265-e3e18ab7bf78" containerID="c6f7efccdc33306d4092875d532c4017a7f1fa6a3d813d210c8c4857a21e6a8c" exitCode=137 Oct 29 23:19:33.093613 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:33.093155 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-c01760" Oct 29 23:19:33.093613 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:33.093221 5057 scope.go:122] "RemoveContainer" containerID="c6f7efccdc33306d4092875d532c4017a7f1fa6a3d813d210c8c4857a21e6a8c" Oct 29 23:19:33.107675 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:33.107656 5057 scope.go:122] "RemoveContainer" containerID="c6f7efccdc33306d4092875d532c4017a7f1fa6a3d813d210c8c4857a21e6a8c" Oct 29 23:19:33.107961 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:19:33.107941 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6f7efccdc33306d4092875d532c4017a7f1fa6a3d813d210c8c4857a21e6a8c\": container with ID starting with c6f7efccdc33306d4092875d532c4017a7f1fa6a3d813d210c8c4857a21e6a8c not found: ID does not exist" containerID="c6f7efccdc33306d4092875d532c4017a7f1fa6a3d813d210c8c4857a21e6a8c" Oct 29 23:19:33.108024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:33.107969 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f7efccdc33306d4092875d532c4017a7f1fa6a3d813d210c8c4857a21e6a8c"} err="failed to get container status \"c6f7efccdc33306d4092875d532c4017a7f1fa6a3d813d210c8c4857a21e6a8c\": rpc error: code = NotFound desc = could not find container \"c6f7efccdc33306d4092875d532c4017a7f1fa6a3d813d210c8c4857a21e6a8c\": container with ID starting with c6f7efccdc33306d4092875d532c4017a7f1fa6a3d813d210c8c4857a21e6a8c not found: ID does not exist" Oct 29 23:19:33.118541 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:19:33.118514 5057 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7896-c01760\" is forbidden: User \"system:node:ip-10-0-135-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-80.ec2.internal' and this object" podUID="c53331ab-bf10-432b-b265-e3e18ab7bf78" pod="opendatahub/test-curl-7896-c01760" Oct 29 23:19:33.398547 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:33.398461 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:33.476405 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:19:33.476363 5057 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7896-c01760\" is forbidden: User \"system:node:ip-10-0-135-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-80.ec2.internal' and this object" podUID="c53331ab-bf10-432b-b265-e3e18ab7bf78" pod="opendatahub/test-curl-7896-c01760" Oct 29 23:19:33.484557 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:33.484530 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53331ab-bf10-432b-b265-e3e18ab7bf78" path="/var/lib/kubelet/pods/c53331ab-bf10-432b-b265-e3e18ab7bf78/volumes" Oct 29 23:19:33.497327 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:19:33.497297 5057 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7896-c01760\" is forbidden: User \"system:node:ip-10-0-135-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-80.ec2.internal' and this object" podUID="c53331ab-bf10-432b-b265-e3e18ab7bf78" pod="opendatahub/test-curl-7896-c01760" Oct 29 23:19:33.986801 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:19:33.986756 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:19:34.104486 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:34.104457 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:34.105380 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:34.105361 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:19:36.386823 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.386783 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2"] Oct 29 23:19:36.391750 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.391721 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2" Oct 29 23:19:36.410960 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.410936 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-e1769a\"" Oct 29 23:19:36.431138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.431112 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2"] Oct 29 23:19:36.465699 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.465666 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh"] Oct 29 23:19:36.470654 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.470628 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh" Oct 29 23:19:36.474068 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.474035 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98609461-cc7d-449e-8dc9-f035a5c53d94-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2\" (UID: \"98609461-cc7d-449e-8dc9-f035a5c53d94\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2" Oct 29 23:19:36.474339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.474320 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxmxn\" (UniqueName: \"kubernetes.io/projected/98609461-cc7d-449e-8dc9-f035a5c53d94-kube-api-access-dxmxn\") pod \"payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2\" (UID: \"98609461-cc7d-449e-8dc9-f035a5c53d94\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2" Oct 29 23:19:36.481640 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.481611 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh"] Oct 29 23:19:36.487840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.487814 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-e1769a-dockercfg-5kmcc\"" Oct 29 23:19:36.635209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.632062 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98609461-cc7d-449e-8dc9-f035a5c53d94-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2\" (UID: \"98609461-cc7d-449e-8dc9-f035a5c53d94\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2" Oct 29 23:19:36.635209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.632217 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxmxn\" (UniqueName: \"kubernetes.io/projected/98609461-cc7d-449e-8dc9-f035a5c53d94-kube-api-access-dxmxn\") pod \"payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2\" (UID: \"98609461-cc7d-449e-8dc9-f035a5c53d94\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2" Oct 29 23:19:36.635209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.632345 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb34f658-47bb-4941-bdca-53d80e885bf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh\" (UID: \"cb34f658-47bb-4941-bdca-53d80e885bf6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh" Oct 29 23:19:36.635209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.632475 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp6d4\" (UniqueName: \"kubernetes.io/projected/cb34f658-47bb-4941-bdca-53d80e885bf6-kube-api-access-jp6d4\") pod \"payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh\" (UID: \"cb34f658-47bb-4941-bdca-53d80e885bf6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh" Oct 29 23:19:36.714275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.713724 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98609461-cc7d-449e-8dc9-f035a5c53d94-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2\" (UID: \"98609461-cc7d-449e-8dc9-f035a5c53d94\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2" Oct 29 23:19:36.739221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.738619 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb34f658-47bb-4941-bdca-53d80e885bf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh\" (UID: \"cb34f658-47bb-4941-bdca-53d80e885bf6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh" Oct 29 23:19:36.739221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.738745 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp6d4\" (UniqueName: \"kubernetes.io/projected/cb34f658-47bb-4941-bdca-53d80e885bf6-kube-api-access-jp6d4\") pod \"payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh\" (UID: \"cb34f658-47bb-4941-bdca-53d80e885bf6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh" Oct 29 23:19:36.757226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.740067 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb34f658-47bb-4941-bdca-53d80e885bf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh\" (UID: \"cb34f658-47bb-4941-bdca-53d80e885bf6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh" Oct 29 23:19:36.773223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.772950 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxmxn\" (UniqueName: \"kubernetes.io/projected/98609461-cc7d-449e-8dc9-f035a5c53d94-kube-api-access-dxmxn\") pod \"payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2\" (UID: \"98609461-cc7d-449e-8dc9-f035a5c53d94\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2" Oct 29 23:19:36.790273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.789693 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp6d4\" (UniqueName: \"kubernetes.io/projected/cb34f658-47bb-4941-bdca-53d80e885bf6-kube-api-access-jp6d4\") pod \"payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh\" (UID: \"cb34f658-47bb-4941-bdca-53d80e885bf6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh" Oct 29 23:19:36.794229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:36.793384 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh" Oct 29 23:19:37.005857 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:37.005740 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2" Oct 29 23:19:37.171088 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:19:37.171060 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb34f658_47bb_4941_bdca_53d80e885bf6.slice/crio-83e7983ab69c02112821d7da80789695cb1728a71a463235e2a1f5b423fb0488 WatchSource:0}: Error finding container 83e7983ab69c02112821d7da80789695cb1728a71a463235e2a1f5b423fb0488: Status 404 returned error can't find the container with id 83e7983ab69c02112821d7da80789695cb1728a71a463235e2a1f5b423fb0488 Oct 29 23:19:37.172244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:37.172217 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh"] Oct 29 23:19:37.315412 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:19:37.315381 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98609461_cc7d_449e_8dc9_f035a5c53d94.slice/crio-875a05f7ca405ac8aae022fca7b40768d2f3bbb9d0fb25a151eda6a7bfe66009 WatchSource:0}: Error finding container 875a05f7ca405ac8aae022fca7b40768d2f3bbb9d0fb25a151eda6a7bfe66009: Status 404 returned error can't find the container with id 875a05f7ca405ac8aae022fca7b40768d2f3bbb9d0fb25a151eda6a7bfe66009 Oct 29 23:19:37.317465 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:37.317436 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2"] Oct 29 23:19:38.155271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:38.155226 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2" event={"ID":"98609461-cc7d-449e-8dc9-f035a5c53d94","Type":"ContainerStarted","Data":"3019edab239b304f9ea2a3907104d17c6b3dc1a22d31eec9e65e0e4f63fdf267"} Oct 29 23:19:38.155792 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:38.155293 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2" event={"ID":"98609461-cc7d-449e-8dc9-f035a5c53d94","Type":"ContainerStarted","Data":"875a05f7ca405ac8aae022fca7b40768d2f3bbb9d0fb25a151eda6a7bfe66009"} Oct 29 23:19:38.155792 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:38.155318 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2" Oct 29 23:19:38.157071 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:38.157039 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh" event={"ID":"cb34f658-47bb-4941-bdca-53d80e885bf6","Type":"ContainerStarted","Data":"be1f3c5bbb82f7e9c09d46575b168ee0b77b94d6ad1b0a968b6b3367708d8159"} Oct 29 23:19:38.157277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:38.157092 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh" event={"ID":"cb34f658-47bb-4941-bdca-53d80e885bf6","Type":"ContainerStarted","Data":"83e7983ab69c02112821d7da80789695cb1728a71a463235e2a1f5b423fb0488"} Oct 29 23:19:38.157277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:38.157120 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh" Oct 29 23:19:38.207927 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:38.207863 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2" podStartSLOduration=2.207842376 podStartE2EDuration="2.207842376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:38.201104646 +0000 UTC m=+2224.852603175" watchObservedRunningTime="2026-10-29 23:19:38.207842376 +0000 UTC m=+2224.859340908" Oct 29 23:19:38.250791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:38.250729 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh" podStartSLOduration=2.250709387 podStartE2EDuration="2.250709387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:38.249228362 +0000 UTC m=+2224.900726884" watchObservedRunningTime="2026-10-29 23:19:38.250709387 +0000 UTC m=+2224.902207921" Oct 29 23:19:43.143790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:43.143756 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm"] Oct 29 23:19:43.148858 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:43.148832 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" Oct 29 23:19:43.192096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:43.192052 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm"] Oct 29 23:19:43.195108 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:43.195081 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-26b322-serving-cert\"" Oct 29 23:19:43.276946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:43.276913 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-maas-api-tls\") pod \"maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm\" (UID: \"10e4f8ea-9a96-4cd7-875e-e9ee710c6515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" Oct 29 23:19:43.277141 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:43.276988 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm\" (UID: \"10e4f8ea-9a96-4cd7-875e-e9ee710c6515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" Oct 29 23:19:43.277141 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:43.277057 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nn2k\" (UniqueName: \"kubernetes.io/projected/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-kube-api-access-6nn2k\") pod \"maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm\" (UID: \"10e4f8ea-9a96-4cd7-875e-e9ee710c6515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" Oct 29 23:19:43.277141 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:43.277109 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-metrics-certs\") pod \"maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm\" (UID: \"10e4f8ea-9a96-4cd7-875e-e9ee710c6515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" Oct 29 23:19:43.378647 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:43.378609 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-maas-api-tls\") pod \"maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm\" (UID: \"10e4f8ea-9a96-4cd7-875e-e9ee710c6515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" Oct 29 23:19:43.378829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:43.378684 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm\" (UID: \"10e4f8ea-9a96-4cd7-875e-e9ee710c6515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" Oct 29 23:19:43.378829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:43.378725 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nn2k\" (UniqueName: \"kubernetes.io/projected/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-kube-api-access-6nn2k\") pod \"maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm\" (UID: \"10e4f8ea-9a96-4cd7-875e-e9ee710c6515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" Oct 29 23:19:43.378829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:43.378789 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-metrics-certs\") pod \"maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm\" (UID: \"10e4f8ea-9a96-4cd7-875e-e9ee710c6515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" Oct 29 23:19:43.379482 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:43.379455 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm\" (UID: \"10e4f8ea-9a96-4cd7-875e-e9ee710c6515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" Oct 29 23:19:43.381240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:43.381204 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-maas-api-tls\") pod \"maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm\" (UID: \"10e4f8ea-9a96-4cd7-875e-e9ee710c6515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" Oct 29 23:19:43.381445 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:43.381421 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-metrics-certs\") pod \"maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm\" (UID: \"10e4f8ea-9a96-4cd7-875e-e9ee710c6515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" Oct 29 23:19:43.439128 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:43.439093 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nn2k\" (UniqueName: \"kubernetes.io/projected/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-kube-api-access-6nn2k\") pod \"maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm\" (UID: \"10e4f8ea-9a96-4cd7-875e-e9ee710c6515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" Oct 29 23:19:43.494157 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:43.494109 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" Oct 29 23:19:43.867770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:43.867725 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm"] Oct 29 23:19:44.208180 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:44.208140 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" event={"ID":"10e4f8ea-9a96-4cd7-875e-e9ee710c6515","Type":"ContainerStarted","Data":"88df9e46e4d881642b687e256425f7316b7e40cb274f662007dd3c244f6e43b2"} Oct 29 23:19:45.281873 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:45.281831 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" event={"ID":"10e4f8ea-9a96-4cd7-875e-e9ee710c6515","Type":"ContainerStarted","Data":"11f8d506b93a4bcc5d437b052ef6387cb606259daabe89bdeabfec7c6b3ad647"} Oct 29 23:19:45.283748 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:45.283707 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" Oct 29 23:19:45.334234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:45.334163 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" podStartSLOduration=1.655725428 podStartE2EDuration="2.334146049s" totalImagesPullingTime="678.420621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:45.329662988 +0000 UTC m=+2231.981161518" watchObservedRunningTime="2026-10-29 23:19:45.334146049 +0000 UTC m=+2231.985644580" Oct 29 23:19:49.168488 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:49.168460 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2" Oct 29 23:19:49.168914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:49.168530 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh" Oct 29 23:19:52.303572 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:19:52.303543 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" Oct 29 23:20:03.915794 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:20:03.915755 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:20:10.393113 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:10.393080 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65"] Oct 29 23:20:10.393753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:10.393315 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" podUID="2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" containerName="istio-proxy" containerID="cri-o://c8dba7952c9d4f9e54635fb77eb4acbfd8f26c7b26c12f5da5b2a66661c14a94" gracePeriod=30 Oct 29 23:20:15.696157 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.696124 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:20:15.794269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.794238 5057 generic.go:356] "Generic (PLEG): container finished" podID="2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" containerID="c8dba7952c9d4f9e54635fb77eb4acbfd8f26c7b26c12f5da5b2a66661c14a94" exitCode=0 Oct 29 23:20:15.794460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.794320 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" event={"ID":"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276","Type":"ContainerDied","Data":"c8dba7952c9d4f9e54635fb77eb4acbfd8f26c7b26c12f5da5b2a66661c14a94"} Oct 29 23:20:15.794460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.794357 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" event={"ID":"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276","Type":"ContainerDied","Data":"5ad2a0fa4a5e8d8cc4ee405ccf112d3fe2327e5f2d02cea4c42919d78de0347d"} Oct 29 23:20:15.794460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.794379 5057 scope.go:122] "RemoveContainer" containerID="c8dba7952c9d4f9e54635fb77eb4acbfd8f26c7b26c12f5da5b2a66661c14a94" Oct 29 23:20:15.794665 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.794516 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65" Oct 29 23:20:15.806089 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.806064 5057 scope.go:122] "RemoveContainer" containerID="1c2aa93d278358cb53967ac1f42e3068758e189c36e5c58fb78d1c8d8c5a6006" Oct 29 23:20:15.820775 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.820739 5057 scope.go:122] "RemoveContainer" containerID="c8dba7952c9d4f9e54635fb77eb4acbfd8f26c7b26c12f5da5b2a66661c14a94" Oct 29 23:20:15.821355 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:20:15.821326 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8dba7952c9d4f9e54635fb77eb4acbfd8f26c7b26c12f5da5b2a66661c14a94\": container with ID starting with c8dba7952c9d4f9e54635fb77eb4acbfd8f26c7b26c12f5da5b2a66661c14a94 not found: ID does not exist" containerID="c8dba7952c9d4f9e54635fb77eb4acbfd8f26c7b26c12f5da5b2a66661c14a94" Oct 29 23:20:15.821473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.821365 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8dba7952c9d4f9e54635fb77eb4acbfd8f26c7b26c12f5da5b2a66661c14a94"} err="failed to get container status \"c8dba7952c9d4f9e54635fb77eb4acbfd8f26c7b26c12f5da5b2a66661c14a94\": rpc error: code = NotFound desc = could not find container \"c8dba7952c9d4f9e54635fb77eb4acbfd8f26c7b26c12f5da5b2a66661c14a94\": container with ID starting with c8dba7952c9d4f9e54635fb77eb4acbfd8f26c7b26c12f5da5b2a66661c14a94 not found: ID does not exist" Oct 29 23:20:15.821473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.821389 5057 scope.go:122] "RemoveContainer" containerID="1c2aa93d278358cb53967ac1f42e3068758e189c36e5c58fb78d1c8d8c5a6006" Oct 29 23:20:15.821705 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:20:15.821682 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2aa93d278358cb53967ac1f42e3068758e189c36e5c58fb78d1c8d8c5a6006\": container with ID starting with 1c2aa93d278358cb53967ac1f42e3068758e189c36e5c58fb78d1c8d8c5a6006 not found: ID does not exist" containerID="1c2aa93d278358cb53967ac1f42e3068758e189c36e5c58fb78d1c8d8c5a6006" Oct 29 23:20:15.821785 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.821711 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2aa93d278358cb53967ac1f42e3068758e189c36e5c58fb78d1c8d8c5a6006"} err="failed to get container status \"1c2aa93d278358cb53967ac1f42e3068758e189c36e5c58fb78d1c8d8c5a6006\": rpc error: code = NotFound desc = could not find container \"1c2aa93d278358cb53967ac1f42e3068758e189c36e5c58fb78d1c8d8c5a6006\": container with ID starting with 1c2aa93d278358cb53967ac1f42e3068758e189c36e5c58fb78d1c8d8c5a6006 not found: ID does not exist" Oct 29 23:20:15.839321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.839295 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-workload-socket\") pod \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " Oct 29 23:20:15.839418 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.839350 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-workload-certs\") pod \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " Oct 29 23:20:15.839418 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.839409 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-credential-socket\") pod \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " Oct 29 23:20:15.839543 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.839462 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istiod-ca-cert\") pod \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " Oct 29 23:20:15.839543 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.839532 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-podinfo\") pod \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " Oct 29 23:20:15.839727 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.839585 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-data\") pod \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " Oct 29 23:20:15.839727 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.839593 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-workload-socket" pod "2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" (UID: "2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:20:15.839727 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.839649 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-kube-api-access-j6zdq\" (UniqueName: \"kubernetes.io/projected/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-kube-api-access-j6zdq\") pod \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " Oct 29 23:20:15.839727 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.839679 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-token\" (UniqueName: \"kubernetes.io/projected/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-token\") pod \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " Oct 29 23:20:15.839727 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.839698 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-workload-certs" pod "2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" (UID: "2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:20:15.840018 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.839784 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-credential-socket" pod "2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" (UID: "2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:20:15.840018 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.839796 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-envoy\") pod \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\" (UID: \"2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276\") " Oct 29 23:20:15.840018 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.839935 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-data" pod "2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" (UID: "2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:20:15.840216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.840155 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istiod-ca-cert" pod "2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" (UID: "2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:20:15.840592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.840571 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:20:15.840663 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.840597 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:20:15.840663 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.840610 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:20:15.840663 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.840624 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:20:15.840663 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.840637 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:20:15.842421 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.842396 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-kube-api-access-j6zdq" pod "2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" (UID: "2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276"). InnerVolumeSpecName "kube-api-access-j6zdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:20:15.842533 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.842433 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-podinfo" pod "2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" (UID: "2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:20:15.842620 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.842597 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-envoy" pod "2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" (UID: "2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:20:15.842686 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.842670 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-token" pod "2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" (UID: "2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:20:15.943001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.942956 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:20:15.943001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.942998 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6zdq\" (UniqueName: \"kubernetes.io/projected/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-kube-api-access-j6zdq\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:20:15.943240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.943014 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:20:15.943240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:15.943026 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:20:16.147519 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:16.147487 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65"] Oct 29 23:20:16.163012 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:16.162979 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-m7j65"] Oct 29 23:20:16.957722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:16.957684 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr"] Oct 29 23:20:16.958680 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:16.958659 5057 cpu_manager.go:414] "containerMap: removing container" podUID="2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" containerName="istio-proxy" Oct 29 23:20:16.958680 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:16.958680 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" containerName="istio-proxy" Oct 29 23:20:16.958865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:16.958770 5057 cpu_manager.go:414] "containerMap: removing container" podUID="2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" containerName="istio-proxy" Oct 29 23:20:16.958865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:16.958779 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" containerName="istio-proxy" Oct 29 23:20:16.959075 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:16.959037 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" containerName="istio-proxy" Oct 29 23:20:16.965266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:16.964474 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" Oct 29 23:20:16.992933 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:16.992903 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-e1769a-serving-cert\"" Oct 29 23:20:17.003427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:17.003304 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr"] Oct 29 23:20:17.060095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:17.060059 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nkgd\" (UniqueName: \"kubernetes.io/projected/f523466e-3408-48cc-aa67-21eec54c0eb0-kube-api-access-9nkgd\") pod \"maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr\" (UID: \"f523466e-3408-48cc-aa67-21eec54c0eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" Oct 29 23:20:17.060332 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:17.060315 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f523466e-3408-48cc-aa67-21eec54c0eb0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr\" (UID: \"f523466e-3408-48cc-aa67-21eec54c0eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" Oct 29 23:20:17.060451 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:17.060433 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f523466e-3408-48cc-aa67-21eec54c0eb0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr\" (UID: \"f523466e-3408-48cc-aa67-21eec54c0eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" Oct 29 23:20:17.060528 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:17.060479 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f523466e-3408-48cc-aa67-21eec54c0eb0-metrics-certs\") pod \"maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr\" (UID: \"f523466e-3408-48cc-aa67-21eec54c0eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" Oct 29 23:20:17.161953 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:17.161920 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nkgd\" (UniqueName: \"kubernetes.io/projected/f523466e-3408-48cc-aa67-21eec54c0eb0-kube-api-access-9nkgd\") pod \"maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr\" (UID: \"f523466e-3408-48cc-aa67-21eec54c0eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" Oct 29 23:20:17.162161 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:17.162106 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f523466e-3408-48cc-aa67-21eec54c0eb0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr\" (UID: \"f523466e-3408-48cc-aa67-21eec54c0eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" Oct 29 23:20:17.162246 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:17.162168 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f523466e-3408-48cc-aa67-21eec54c0eb0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr\" (UID: \"f523466e-3408-48cc-aa67-21eec54c0eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" Oct 29 23:20:17.162246 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:17.162230 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f523466e-3408-48cc-aa67-21eec54c0eb0-metrics-certs\") pod \"maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr\" (UID: \"f523466e-3408-48cc-aa67-21eec54c0eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" Oct 29 23:20:17.162993 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:17.162945 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f523466e-3408-48cc-aa67-21eec54c0eb0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr\" (UID: \"f523466e-3408-48cc-aa67-21eec54c0eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" Oct 29 23:20:17.165107 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:17.165043 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f523466e-3408-48cc-aa67-21eec54c0eb0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr\" (UID: \"f523466e-3408-48cc-aa67-21eec54c0eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" Oct 29 23:20:17.165107 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:17.165052 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f523466e-3408-48cc-aa67-21eec54c0eb0-metrics-certs\") pod \"maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr\" (UID: \"f523466e-3408-48cc-aa67-21eec54c0eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" Oct 29 23:20:17.188931 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:17.188903 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nkgd\" (UniqueName: \"kubernetes.io/projected/f523466e-3408-48cc-aa67-21eec54c0eb0-kube-api-access-9nkgd\") pod \"maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr\" (UID: \"f523466e-3408-48cc-aa67-21eec54c0eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" Oct 29 23:20:17.296017 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:17.293310 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" Oct 29 23:20:17.485149 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:17.485113 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" path="/var/lib/kubelet/pods/2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276/volumes" Oct 29 23:20:17.638684 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:20:17.638652 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf523466e_3408_48cc_aa67_21eec54c0eb0.slice/crio-df0144c7b108fba061a88401a6455c3f46233d5518f74b23aa17e53dd9d82fbc WatchSource:0}: Error finding container df0144c7b108fba061a88401a6455c3f46233d5518f74b23aa17e53dd9d82fbc: Status 404 returned error can't find the container with id df0144c7b108fba061a88401a6455c3f46233d5518f74b23aa17e53dd9d82fbc Oct 29 23:20:17.639583 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:17.639557 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr"] Oct 29 23:20:17.810712 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:17.810677 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" event={"ID":"f523466e-3408-48cc-aa67-21eec54c0eb0","Type":"ContainerStarted","Data":"df0144c7b108fba061a88401a6455c3f46233d5518f74b23aa17e53dd9d82fbc"} Oct 29 23:20:18.820066 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:18.820034 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" event={"ID":"f523466e-3408-48cc-aa67-21eec54c0eb0","Type":"ContainerStarted","Data":"6444cb5cfbcfb421c80865814210ae16b4bb5b1fd8af60a21f1229a71e4f3e09"} Oct 29 23:20:18.820582 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:18.820096 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" Oct 29 23:20:18.866222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:18.866032 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" podStartSLOduration=2.5492105 podStartE2EDuration="2.866016466s" totalImagesPullingTime="316.805966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:18.865757914 +0000 UTC m=+2265.517256445" watchObservedRunningTime="2026-10-29 23:20:18.866016466 +0000 UTC m=+2265.517514999" Oct 29 23:20:24.858261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:24.857399 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" Oct 29 23:20:30.232374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:30.232329 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:20:30.232883 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:30.232424 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" podUID="fa2b610a-d82f-47f8-93cd-736a51e1d52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:20:33.049048 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:33.049003 5057 generic.go:356] "Generic (PLEG): container finished" podID="fa2b610a-d82f-47f8-93cd-736a51e1d52e" containerID="7f391a6bddc7cb5aa23ac88ae5dfa48dfdee05eb3e2f6a74adeac80c89a97845" exitCode=0 Oct 29 23:20:33.049628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:33.049067 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" event={"ID":"fa2b610a-d82f-47f8-93cd-736a51e1d52e","Type":"ContainerDied","Data":"7f391a6bddc7cb5aa23ac88ae5dfa48dfdee05eb3e2f6a74adeac80c89a97845"} Oct 29 23:20:33.049628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:33.049561 5057 scope.go:122] "RemoveContainer" containerID="7f391a6bddc7cb5aa23ac88ae5dfa48dfdee05eb3e2f6a74adeac80c89a97845" Oct 29 23:20:33.051864 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:33.051823 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:20:33.051992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:33.051915 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:20:33.051992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:33.051963 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:20:33.919448 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:20:33.919404 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:20:34.077390 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:34.077353 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" event={"ID":"fa2b610a-d82f-47f8-93cd-736a51e1d52e","Type":"ContainerStarted","Data":"36751214715a146ae5962f7872a2d454c106c56437a8dd8fe5149d8ce96fa8c7"} Oct 29 23:20:35.046213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:35.045528 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:20:35.046213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:35.045691 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:20:35.046213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:35.045749 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" podUID="fa2b610a-d82f-47f8-93cd-736a51e1d52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 29 23:20:36.045210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:36.045095 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:20:36.045210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:36.045154 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" podUID="fa2b610a-d82f-47f8-93cd-736a51e1d52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 29 23:20:37.043210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:37.038505 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:20:37.043210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:37.038589 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" podUID="fa2b610a-d82f-47f8-93cd-736a51e1d52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 29 23:20:37.456247 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:37.445358 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:20:38.039618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:38.039583 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:20:38.040629 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:38.040595 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" podUID="fa2b610a-d82f-47f8-93cd-736a51e1d52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 29 23:20:39.038543 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:39.038504 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:20:39.039179 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:39.039144 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" podUID="fa2b610a-d82f-47f8-93cd-736a51e1d52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 29 23:20:40.057209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:40.057156 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:20:40.078213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:40.060122 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" podUID="fa2b610a-d82f-47f8-93cd-736a51e1d52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 29 23:20:41.084240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:41.084205 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:20:41.084846 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:41.084806 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:20:41.085584 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:41.085564 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:20:50.697111 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:50.697078 5057 generic.go:356] "Generic (PLEG): container finished" podID="d89378ad-ddab-4899-9041-7c4cabafa109" containerID="38f1f62e9e3c5f520f43a26031fde07bebe829a1260e36b8ac12b46fb3d650d0" exitCode=0 Oct 29 23:20:50.697581 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:50.697135 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" event={"ID":"d89378ad-ddab-4899-9041-7c4cabafa109","Type":"ContainerDied","Data":"38f1f62e9e3c5f520f43a26031fde07bebe829a1260e36b8ac12b46fb3d650d0"} Oct 29 23:20:50.697636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:50.697616 5057 scope.go:122] "RemoveContainer" containerID="38f1f62e9e3c5f520f43a26031fde07bebe829a1260e36b8ac12b46fb3d650d0" Oct 29 23:20:50.699519 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:50.699489 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:20:50.699590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:50.699560 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:20:50.699627 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:50.699592 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:20:51.707872 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:51.707839 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" event={"ID":"d89378ad-ddab-4899-9041-7c4cabafa109","Type":"ContainerStarted","Data":"4b0869b5d5ac71822fff422c7c14f9e1f26b1f3369a4beefde36cf8eef225651"} Oct 29 23:20:51.716117 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:51.716089 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:20:51.717285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:51.717249 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 29 23:20:51.717423 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:51.717304 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" podUID="d89378ad-ddab-4899-9041-7c4cabafa109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 29 23:20:52.716371 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:52.716338 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 29 23:20:52.716789 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:52.716391 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" podUID="d89378ad-ddab-4899-9041-7c4cabafa109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 29 23:20:53.716814 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:53.716779 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 29 23:20:53.717266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:53.716842 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" podUID="d89378ad-ddab-4899-9041-7c4cabafa109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 29 23:20:54.716250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:54.716164 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 29 23:20:54.716472 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:54.716251 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" podUID="d89378ad-ddab-4899-9041-7c4cabafa109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 29 23:20:55.716764 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:55.716731 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 29 23:20:55.717150 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:55.716788 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" podUID="d89378ad-ddab-4899-9041-7c4cabafa109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 29 23:20:55.927806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:55.927768 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:20:56.716789 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:56.716751 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 29 23:20:56.717227 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:56.716816 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" podUID="d89378ad-ddab-4899-9041-7c4cabafa109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 29 23:20:57.721362 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:57.721332 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:20:57.722629 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:57.722606 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:20:57.778221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:57.778175 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc"] Oct 29 23:20:57.779574 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:57.779553 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ecdcde5-2bbb-42cc-b89e-2ff1e9da5276" containerName="istio-proxy" Oct 29 23:20:57.783807 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:57.783769 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:57.816330 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:57.816291 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc"] Oct 29 23:20:57.900257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:57.898859 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlxxh\" (UniqueName: \"kubernetes.io/projected/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-kube-api-access-dlxxh\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:57.900257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:57.898921 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-workload-certs\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:57.900257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:57.899054 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-workload-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:57.900257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:57.899152 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-data\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:57.900257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:57.899225 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istiod-ca-cert\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:57.900257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:57.899268 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-credential-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:57.900257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:57.899431 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-envoy\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:57.900257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:57.899610 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-token\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:57.900257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:57.899737 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-podinfo\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:58.001298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.001208 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-envoy\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:58.001482 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.001331 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-token\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:58.001482 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.001405 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-podinfo\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:58.001482 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.001447 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlxxh\" (UniqueName: \"kubernetes.io/projected/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-kube-api-access-dlxxh\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:58.001482 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.001475 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-workload-certs\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:58.001706 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.001515 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-workload-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:58.001706 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.001585 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-data\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:58.001706 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.001634 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istiod-ca-cert\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:58.001706 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.001671 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-credential-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:58.001950 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.001922 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-workload-certs\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:58.002104 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.002082 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-credential-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:58.002229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.002126 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-data\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:58.002344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.002327 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-workload-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:58.002923 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.002897 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istiod-ca-cert\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:58.004398 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.004372 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-envoy\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:58.004510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.004452 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-podinfo\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:58.028905 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.028875 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlxxh\" (UniqueName: \"kubernetes.io/projected/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-kube-api-access-dlxxh\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:58.029029 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.028961 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-token\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:58.101824 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.101789 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:58.428881 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:20:58.428843 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f8ebdf_e0d2_4594_8d4c_9a177233e48a.slice/crio-2c5e5883a745aafec615cda475acd46c974b6e0e82d801c406308900d958bef5 WatchSource:0}: Error finding container 2c5e5883a745aafec615cda475acd46c974b6e0e82d801c406308900d958bef5: Status 404 returned error can't find the container with id 2c5e5883a745aafec615cda475acd46c974b6e0e82d801c406308900d958bef5 Oct 29 23:20:58.431282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.431255 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc"] Oct 29 23:20:58.432138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.432103 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:20:58.432220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.432174 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:20:58.432274 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.432231 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:20:58.793857 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.793764 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" event={"ID":"36f8ebdf-e0d2-4594-8d4c-9a177233e48a","Type":"ContainerStarted","Data":"37d970c67bf8317c74effd04b402073cac3f9b2d6b96ecd2da122c8d47cfd900"} Oct 29 23:20:58.793857 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.793831 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" event={"ID":"36f8ebdf-e0d2-4594-8d4c-9a177233e48a","Type":"ContainerStarted","Data":"2c5e5883a745aafec615cda475acd46c974b6e0e82d801c406308900d958bef5"} Oct 29 23:20:58.857606 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:58.857556 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" podStartSLOduration=1.8575410749999999 podStartE2EDuration="1.857541075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:58.844755399 +0000 UTC m=+2305.496253919" watchObservedRunningTime="2026-10-29 23:20:58.857541075 +0000 UTC m=+2305.509039604" Oct 29 23:20:59.102588 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:59.102500 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:20:59.104440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:59.104408 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:20:59.104585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:20:59.104456 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" podUID="36f8ebdf-e0d2-4594-8d4c-9a177233e48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 29 23:21:00.102925 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:00.102890 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:21:00.104260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:00.103033 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" podUID="36f8ebdf-e0d2-4594-8d4c-9a177233e48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 29 23:21:01.103001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:01.102967 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:21:01.103455 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:01.103015 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" podUID="36f8ebdf-e0d2-4594-8d4c-9a177233e48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 29 23:21:02.102405 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:02.102372 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:21:02.102629 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:02.102432 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" podUID="36f8ebdf-e0d2-4594-8d4c-9a177233e48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 29 23:21:03.104657 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:03.104616 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:21:03.105291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:03.105255 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" podUID="36f8ebdf-e0d2-4594-8d4c-9a177233e48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 29 23:21:03.915884 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:21:03.915819 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:21:04.102699 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:04.102668 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:21:04.102906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:04.102722 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" podUID="36f8ebdf-e0d2-4594-8d4c-9a177233e48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 29 23:21:05.102206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:05.102161 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:21:05.102627 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:05.102266 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" podUID="36f8ebdf-e0d2-4594-8d4c-9a177233e48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 29 23:21:06.107126 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:06.107094 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:21:06.107622 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:06.107604 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:21:06.108505 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:06.108484 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:21:10.663571 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:10.663538 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc"] Oct 29 23:21:10.664084 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:10.663714 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" podUID="36f8ebdf-e0d2-4594-8d4c-9a177233e48a" containerName="istio-proxy" containerID="cri-o://37d970c67bf8317c74effd04b402073cac3f9b2d6b96ecd2da122c8d47cfd900" gracePeriod=30 Oct 29 23:21:10.677116 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:10.677081 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm"] Oct 29 23:21:10.677389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:10.677359 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" podUID="fa2b610a-d82f-47f8-93cd-736a51e1d52e" containerName="istio-proxy" containerID="cri-o://36751214715a146ae5962f7872a2d454c106c56437a8dd8fe5149d8ce96fa8c7" gracePeriod=30 Oct 29 23:21:11.092246 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:11.091302 5057 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:21:11.092246 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:11.091387 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" podUID="fa2b610a-d82f-47f8-93cd-736a51e1d52e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:21:13.545492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:13.545455 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw"] Oct 29 23:21:13.551495 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:13.551471 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw" Oct 29 23:21:13.576091 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:13.576063 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw"] Oct 29 23:21:13.742225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:13.611765 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-206095\"" Oct 29 23:21:13.742225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:13.631396 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-206095-dockercfg-h9g5v\"" Oct 29 23:21:13.742225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:13.659579 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8785n\" (UniqueName: \"kubernetes.io/projected/41da29c2-0216-4adf-88ba-32d655e88efd-kube-api-access-8785n\") pod \"payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw\" (UID: \"41da29c2-0216-4adf-88ba-32d655e88efd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw" Oct 29 23:21:13.742225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:13.698107 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41da29c2-0216-4adf-88ba-32d655e88efd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw\" (UID: \"41da29c2-0216-4adf-88ba-32d655e88efd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw" Oct 29 23:21:13.819317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:13.812516 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8785n\" (UniqueName: \"kubernetes.io/projected/41da29c2-0216-4adf-88ba-32d655e88efd-kube-api-access-8785n\") pod \"payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw\" (UID: \"41da29c2-0216-4adf-88ba-32d655e88efd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw" Oct 29 23:21:13.819317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:13.812595 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41da29c2-0216-4adf-88ba-32d655e88efd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw\" (UID: \"41da29c2-0216-4adf-88ba-32d655e88efd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw" Oct 29 23:21:13.819317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:13.813828 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41da29c2-0216-4adf-88ba-32d655e88efd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw\" (UID: \"41da29c2-0216-4adf-88ba-32d655e88efd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw" Oct 29 23:21:13.926534 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:13.926164 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8785n\" (UniqueName: \"kubernetes.io/projected/41da29c2-0216-4adf-88ba-32d655e88efd-kube-api-access-8785n\") pod \"payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw\" (UID: \"41da29c2-0216-4adf-88ba-32d655e88efd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw" Oct 29 23:21:14.168389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:14.168311 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw" Oct 29 23:21:14.537879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:14.537842 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw"] Oct 29 23:21:14.546488 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:21:14.541991 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41da29c2_0216_4adf_88ba_32d655e88efd.slice/crio-8726510575d9a0134d5748d34456f48e6e01785e5f5156001fb2b1ab6233d8e7 WatchSource:0}: Error finding container 8726510575d9a0134d5748d34456f48e6e01785e5f5156001fb2b1ab6233d8e7: Status 404 returned error can't find the container with id 8726510575d9a0134d5748d34456f48e6e01785e5f5156001fb2b1ab6233d8e7 Oct 29 23:21:15.130353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:15.130312 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw" event={"ID":"41da29c2-0216-4adf-88ba-32d655e88efd","Type":"ContainerStarted","Data":"3dcbc601edc351292874c97858f175ece0229ad46ca850b54b13bb63cb3bfc6f"} Oct 29 23:21:15.130553 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:15.130378 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw" event={"ID":"41da29c2-0216-4adf-88ba-32d655e88efd","Type":"ContainerStarted","Data":"8726510575d9a0134d5748d34456f48e6e01785e5f5156001fb2b1ab6233d8e7"} Oct 29 23:21:15.130553 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:15.130457 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw" Oct 29 23:21:15.177412 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:15.177361 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw" podStartSLOduration=2.177343402 podStartE2EDuration="2.177343402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:15.174356859 +0000 UTC m=+2321.825855391" watchObservedRunningTime="2026-10-29 23:21:15.177343402 +0000 UTC m=+2321.828841933" Oct 29 23:21:15.841065 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:21:15.841024 5057 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2b610a_d82f_47f8_93cd_736a51e1d52e.slice/crio-36751214715a146ae5962f7872a2d454c106c56437a8dd8fe5149d8ce96fa8c7.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:21:16.027617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.027583 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:21:16.067678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.067650 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:21:16.167221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.167137 5057 generic.go:356] "Generic (PLEG): container finished" podID="36f8ebdf-e0d2-4594-8d4c-9a177233e48a" containerID="37d970c67bf8317c74effd04b402073cac3f9b2d6b96ecd2da122c8d47cfd900" exitCode=0 Oct 29 23:21:16.167425 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.167354 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" Oct 29 23:21:16.167534 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.167508 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" event={"ID":"36f8ebdf-e0d2-4594-8d4c-9a177233e48a","Type":"ContainerDied","Data":"37d970c67bf8317c74effd04b402073cac3f9b2d6b96ecd2da122c8d47cfd900"} Oct 29 23:21:16.167612 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.167599 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc" event={"ID":"36f8ebdf-e0d2-4594-8d4c-9a177233e48a","Type":"ContainerDied","Data":"2c5e5883a745aafec615cda475acd46c974b6e0e82d801c406308900d958bef5"} Oct 29 23:21:16.167677 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.167638 5057 scope.go:122] "RemoveContainer" containerID="37d970c67bf8317c74effd04b402073cac3f9b2d6b96ecd2da122c8d47cfd900" Oct 29 23:21:16.171278 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.171256 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istiod-ca-cert\") pod \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " Oct 29 23:21:16.171387 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.171303 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-credential-socket\") pod \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " Oct 29 23:21:16.171387 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.171335 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-podinfo\") pod \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " Oct 29 23:21:16.171387 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.171367 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-workload-certs\") pod \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " Oct 29 23:21:16.171573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.171406 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2b610a-d82f-47f8-93cd-736a51e1d52e-kube-api-access-hflnr\" (UniqueName: \"kubernetes.io/projected/fa2b610a-d82f-47f8-93cd-736a51e1d52e-kube-api-access-hflnr\") pod \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " Oct 29 23:21:16.171573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.171427 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-workload-certs\") pod \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " Oct 29 23:21:16.171573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.171453 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istiod-ca-cert\") pod \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " Oct 29 23:21:16.171573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.171496 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-podinfo\") pod \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " Oct 29 23:21:16.171573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.171522 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-credential-socket\") pod \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " Oct 29 23:21:16.171573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.171554 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-kube-api-access-dlxxh\" (UniqueName: \"kubernetes.io/projected/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-kube-api-access-dlxxh\") pod \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " Oct 29 23:21:16.171898 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.171595 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-data\") pod \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " Oct 29 23:21:16.171898 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.171670 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-workload-socket\") pod \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " Oct 29 23:21:16.171898 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.171694 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-envoy\") pod \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " Oct 29 23:21:16.171898 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.171717 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-data\") pod \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " Oct 29 23:21:16.171898 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.171764 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-token\" (UniqueName: \"kubernetes.io/projected/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-token\") pod \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " Oct 29 23:21:16.171898 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.171798 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-envoy\") pod \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " Oct 29 23:21:16.174200 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.173718 5057 generic.go:356] "Generic (PLEG): container finished" podID="fa2b610a-d82f-47f8-93cd-736a51e1d52e" containerID="36751214715a146ae5962f7872a2d454c106c56437a8dd8fe5149d8ce96fa8c7" exitCode=0 Oct 29 23:21:16.174474 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.174448 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-workload-certs" pod "fa2b610a-d82f-47f8-93cd-736a51e1d52e" (UID: "fa2b610a-d82f-47f8-93cd-736a51e1d52e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:16.174990 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.174970 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istiod-ca-cert" pod "fa2b610a-d82f-47f8-93cd-736a51e1d52e" (UID: "fa2b610a-d82f-47f8-93cd-736a51e1d52e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:21:16.175402 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.175381 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-credential-socket" pod "36f8ebdf-e0d2-4594-8d4c-9a177233e48a" (UID: "36f8ebdf-e0d2-4594-8d4c-9a177233e48a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:16.176010 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.175993 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" Oct 29 23:21:16.176300 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.176276 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" event={"ID":"fa2b610a-d82f-47f8-93cd-736a51e1d52e","Type":"ContainerDied","Data":"36751214715a146ae5962f7872a2d454c106c56437a8dd8fe5149d8ce96fa8c7"} Oct 29 23:21:16.176462 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.176442 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm" event={"ID":"fa2b610a-d82f-47f8-93cd-736a51e1d52e","Type":"ContainerDied","Data":"0dabd74b92673ac2d89e358fb49b120b8eb3d99d60ecfe11f149336244411b33"} Oct 29 23:21:16.176967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.176944 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-workload-socket" pod "36f8ebdf-e0d2-4594-8d4c-9a177233e48a" (UID: "36f8ebdf-e0d2-4594-8d4c-9a177233e48a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:16.177293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.177273 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-credential-socket" pod "fa2b610a-d82f-47f8-93cd-736a51e1d52e" (UID: "fa2b610a-d82f-47f8-93cd-736a51e1d52e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:16.178315 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.178291 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-podinfo" pod "36f8ebdf-e0d2-4594-8d4c-9a177233e48a" (UID: "36f8ebdf-e0d2-4594-8d4c-9a177233e48a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:21:16.179298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.179275 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-podinfo" pod "fa2b610a-d82f-47f8-93cd-736a51e1d52e" (UID: "fa2b610a-d82f-47f8-93cd-736a51e1d52e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:21:16.184503 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.179806 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-workload-certs" pod "36f8ebdf-e0d2-4594-8d4c-9a177233e48a" (UID: "36f8ebdf-e0d2-4594-8d4c-9a177233e48a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.184781 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-token" pod "fa2b610a-d82f-47f8-93cd-736a51e1d52e" (UID: "fa2b610a-d82f-47f8-93cd-736a51e1d52e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.185072 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-data" pod "fa2b610a-d82f-47f8-93cd-736a51e1d52e" (UID: "fa2b610a-d82f-47f8-93cd-736a51e1d52e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.185935 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-envoy" pod "36f8ebdf-e0d2-4594-8d4c-9a177233e48a" (UID: "36f8ebdf-e0d2-4594-8d4c-9a177233e48a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.186241 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istiod-ca-cert" pod "36f8ebdf-e0d2-4594-8d4c-9a177233e48a" (UID: "36f8ebdf-e0d2-4594-8d4c-9a177233e48a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.186435 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-data" pod "36f8ebdf-e0d2-4594-8d4c-9a177233e48a" (UID: "36f8ebdf-e0d2-4594-8d4c-9a177233e48a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.188628 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2b610a-d82f-47f8-93cd-736a51e1d52e-kube-api-access-hflnr" pod "fa2b610a-d82f-47f8-93cd-736a51e1d52e" (UID: "fa2b610a-d82f-47f8-93cd-736a51e1d52e"). InnerVolumeSpecName "kube-api-access-hflnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.188947 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-workload-socket" pod "fa2b610a-d82f-47f8-93cd-736a51e1d52e" (UID: "fa2b610a-d82f-47f8-93cd-736a51e1d52e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.188771 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-workload-socket\") pod \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\" (UID: \"fa2b610a-d82f-47f8-93cd-736a51e1d52e\") " Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.189065 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-token\" (UniqueName: \"kubernetes.io/projected/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-token\") pod \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\" (UID: \"36f8ebdf-e0d2-4594-8d4c-9a177233e48a\") " Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.189822 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.189843 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.189857 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hflnr\" (UniqueName: \"kubernetes.io/projected/fa2b610a-d82f-47f8-93cd-736a51e1d52e-kube-api-access-hflnr\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.189871 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.189884 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.189895 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.189908 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.189921 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.189950 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.189962 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.189974 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.189986 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.189998 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.190010 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.190023 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.193577 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-envoy" pod "fa2b610a-d82f-47f8-93cd-736a51e1d52e" (UID: "fa2b610a-d82f-47f8-93cd-736a51e1d52e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:16.194280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.193583 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-kube-api-access-dlxxh" pod "36f8ebdf-e0d2-4594-8d4c-9a177233e48a" (UID: "36f8ebdf-e0d2-4594-8d4c-9a177233e48a"). InnerVolumeSpecName "kube-api-access-dlxxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:16.195733 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.195714 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-token" pod "36f8ebdf-e0d2-4594-8d4c-9a177233e48a" (UID: "36f8ebdf-e0d2-4594-8d4c-9a177233e48a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:16.199849 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.199831 5057 scope.go:122] "RemoveContainer" containerID="37d970c67bf8317c74effd04b402073cac3f9b2d6b96ecd2da122c8d47cfd900" Oct 29 23:21:16.200423 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:21:16.200362 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d970c67bf8317c74effd04b402073cac3f9b2d6b96ecd2da122c8d47cfd900\": container with ID starting with 37d970c67bf8317c74effd04b402073cac3f9b2d6b96ecd2da122c8d47cfd900 not found: ID does not exist" containerID="37d970c67bf8317c74effd04b402073cac3f9b2d6b96ecd2da122c8d47cfd900" Oct 29 23:21:16.200423 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.200393 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d970c67bf8317c74effd04b402073cac3f9b2d6b96ecd2da122c8d47cfd900"} err="failed to get container status \"37d970c67bf8317c74effd04b402073cac3f9b2d6b96ecd2da122c8d47cfd900\": rpc error: code = NotFound desc = could not find container \"37d970c67bf8317c74effd04b402073cac3f9b2d6b96ecd2da122c8d47cfd900\": container with ID starting with 37d970c67bf8317c74effd04b402073cac3f9b2d6b96ecd2da122c8d47cfd900 not found: ID does not exist" Oct 29 23:21:16.200423 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.200411 5057 scope.go:122] "RemoveContainer" containerID="36751214715a146ae5962f7872a2d454c106c56437a8dd8fe5149d8ce96fa8c7" Oct 29 23:21:16.216552 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.215982 5057 scope.go:122] "RemoveContainer" containerID="7f391a6bddc7cb5aa23ac88ae5dfa48dfdee05eb3e2f6a74adeac80c89a97845" Oct 29 23:21:16.229028 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.228987 5057 scope.go:122] "RemoveContainer" containerID="36751214715a146ae5962f7872a2d454c106c56437a8dd8fe5149d8ce96fa8c7" Oct 29 23:21:16.229419 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:21:16.229381 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36751214715a146ae5962f7872a2d454c106c56437a8dd8fe5149d8ce96fa8c7\": container with ID starting with 36751214715a146ae5962f7872a2d454c106c56437a8dd8fe5149d8ce96fa8c7 not found: ID does not exist" containerID="36751214715a146ae5962f7872a2d454c106c56437a8dd8fe5149d8ce96fa8c7" Oct 29 23:21:16.229534 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.229422 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36751214715a146ae5962f7872a2d454c106c56437a8dd8fe5149d8ce96fa8c7"} err="failed to get container status \"36751214715a146ae5962f7872a2d454c106c56437a8dd8fe5149d8ce96fa8c7\": rpc error: code = NotFound desc = could not find container \"36751214715a146ae5962f7872a2d454c106c56437a8dd8fe5149d8ce96fa8c7\": container with ID starting with 36751214715a146ae5962f7872a2d454c106c56437a8dd8fe5149d8ce96fa8c7 not found: ID does not exist" Oct 29 23:21:16.229534 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.229447 5057 scope.go:122] "RemoveContainer" containerID="7f391a6bddc7cb5aa23ac88ae5dfa48dfdee05eb3e2f6a74adeac80c89a97845" Oct 29 23:21:16.229775 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:21:16.229753 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f391a6bddc7cb5aa23ac88ae5dfa48dfdee05eb3e2f6a74adeac80c89a97845\": container with ID starting with 7f391a6bddc7cb5aa23ac88ae5dfa48dfdee05eb3e2f6a74adeac80c89a97845 not found: ID does not exist" containerID="7f391a6bddc7cb5aa23ac88ae5dfa48dfdee05eb3e2f6a74adeac80c89a97845" Oct 29 23:21:16.229847 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.229783 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f391a6bddc7cb5aa23ac88ae5dfa48dfdee05eb3e2f6a74adeac80c89a97845"} err="failed to get container status \"7f391a6bddc7cb5aa23ac88ae5dfa48dfdee05eb3e2f6a74adeac80c89a97845\": rpc error: code = NotFound desc = could not find container \"7f391a6bddc7cb5aa23ac88ae5dfa48dfdee05eb3e2f6a74adeac80c89a97845\": container with ID starting with 7f391a6bddc7cb5aa23ac88ae5dfa48dfdee05eb3e2f6a74adeac80c89a97845 not found: ID does not exist" Oct 29 23:21:16.291669 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.291590 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.291669 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.291621 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlxxh\" (UniqueName: \"kubernetes.io/projected/36f8ebdf-e0d2-4594-8d4c-9a177233e48a-kube-api-access-dlxxh\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.291669 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.291631 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2b610a-d82f-47f8-93cd-736a51e1d52e-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.522372 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.522336 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc"] Oct 29 23:21:16.536528 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.536503 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-b5szc"] Oct 29 23:21:16.583042 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.582970 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm"] Oct 29 23:21:16.586897 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:16.586871 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-dnffm"] Oct 29 23:21:17.483558 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:17.483525 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f8ebdf-e0d2-4594-8d4c-9a177233e48a" path="/var/lib/kubelet/pods/36f8ebdf-e0d2-4594-8d4c-9a177233e48a/volumes" Oct 29 23:21:17.484055 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:17.484039 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa2b610a-d82f-47f8-93cd-736a51e1d52e" path="/var/lib/kubelet/pods/fa2b610a-d82f-47f8-93cd-736a51e1d52e/volumes" Oct 29 23:21:21.561213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:21.561167 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5"] Oct 29 23:21:21.561729 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:21.561481 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" podUID="4cc22e58-ddce-4866-b17d-8672d775fc4a" containerName="istio-proxy" containerID="cri-o://a8c12da6975a29ad7545bec1427b4689b37026be94b62c4ac2af49f3f2e02053" gracePeriod=30 Oct 29 23:21:26.179431 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:26.179404 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw" Oct 29 23:21:26.940304 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:26.940274 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:21:27.007644 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.007611 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-workload-certs\") pod \"4cc22e58-ddce-4866-b17d-8672d775fc4a\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " Oct 29 23:21:27.007858 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.007718 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-credential-socket\") pod \"4cc22e58-ddce-4866-b17d-8672d775fc4a\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " Oct 29 23:21:27.007858 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.007820 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-podinfo\") pod \"4cc22e58-ddce-4866-b17d-8672d775fc4a\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " Oct 29 23:21:27.007995 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.007875 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-data\") pod \"4cc22e58-ddce-4866-b17d-8672d775fc4a\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " Oct 29 23:21:27.007995 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.007930 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-workload-certs" pod "4cc22e58-ddce-4866-b17d-8672d775fc4a" (UID: "4cc22e58-ddce-4866-b17d-8672d775fc4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:27.007995 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.007978 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cc22e58-ddce-4866-b17d-8672d775fc4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc22e58-ddce-4866-b17d-8672d775fc4a-istiod-ca-cert\") pod \"4cc22e58-ddce-4866-b17d-8672d775fc4a\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " Oct 29 23:21:27.008176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.008119 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-envoy\") pod \"4cc22e58-ddce-4866-b17d-8672d775fc4a\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " Oct 29 23:21:27.008176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.008129 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-credential-socket" pod "4cc22e58-ddce-4866-b17d-8672d775fc4a" (UID: "4cc22e58-ddce-4866-b17d-8672d775fc4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:27.008176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.008155 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc22e58-ddce-4866-b17d-8672d775fc4a-kube-api-access-jrnnc\" (UniqueName: \"kubernetes.io/projected/4cc22e58-ddce-4866-b17d-8672d775fc4a-kube-api-access-jrnnc\") pod \"4cc22e58-ddce-4866-b17d-8672d775fc4a\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " Oct 29 23:21:27.008474 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.008210 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-token\" (UniqueName: \"kubernetes.io/projected/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-token\") pod \"4cc22e58-ddce-4866-b17d-8672d775fc4a\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " Oct 29 23:21:27.008474 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.008253 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-workload-socket\") pod \"4cc22e58-ddce-4866-b17d-8672d775fc4a\" (UID: \"4cc22e58-ddce-4866-b17d-8672d775fc4a\") " Oct 29 23:21:27.008474 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.008357 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-data" pod "4cc22e58-ddce-4866-b17d-8672d775fc4a" (UID: "4cc22e58-ddce-4866-b17d-8672d775fc4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:27.008989 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.008970 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:27.009085 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.008993 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:27.009085 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.009006 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:27.009305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.009283 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-workload-socket" pod "4cc22e58-ddce-4866-b17d-8672d775fc4a" (UID: "4cc22e58-ddce-4866-b17d-8672d775fc4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:27.009761 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.009702 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc22e58-ddce-4866-b17d-8672d775fc4a-istiod-ca-cert" pod "4cc22e58-ddce-4866-b17d-8672d775fc4a" (UID: "4cc22e58-ddce-4866-b17d-8672d775fc4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:21:27.011462 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.011436 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-podinfo" pod "4cc22e58-ddce-4866-b17d-8672d775fc4a" (UID: "4cc22e58-ddce-4866-b17d-8672d775fc4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:21:27.011595 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.011575 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc22e58-ddce-4866-b17d-8672d775fc4a-kube-api-access-jrnnc" pod "4cc22e58-ddce-4866-b17d-8672d775fc4a" (UID: "4cc22e58-ddce-4866-b17d-8672d775fc4a"). InnerVolumeSpecName "kube-api-access-jrnnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:27.011668 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.011637 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-token" pod "4cc22e58-ddce-4866-b17d-8672d775fc4a" (UID: "4cc22e58-ddce-4866-b17d-8672d775fc4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:27.011915 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.011892 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-envoy" pod "4cc22e58-ddce-4866-b17d-8672d775fc4a" (UID: "4cc22e58-ddce-4866-b17d-8672d775fc4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:27.116753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.111544 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:27.116753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.111591 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc22e58-ddce-4866-b17d-8672d775fc4a-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:27.116753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.111606 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:27.116753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.111621 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrnnc\" (UniqueName: \"kubernetes.io/projected/4cc22e58-ddce-4866-b17d-8672d775fc4a-kube-api-access-jrnnc\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:27.116753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.111636 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc22e58-ddce-4866-b17d-8672d775fc4a-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:27.116753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.111652 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc22e58-ddce-4866-b17d-8672d775fc4a-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:27.286789 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.286755 5057 generic.go:356] "Generic (PLEG): container finished" podID="4cc22e58-ddce-4866-b17d-8672d775fc4a" containerID="a8c12da6975a29ad7545bec1427b4689b37026be94b62c4ac2af49f3f2e02053" exitCode=0 Oct 29 23:21:27.287358 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.286804 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" Oct 29 23:21:27.287358 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.286846 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" event={"ID":"4cc22e58-ddce-4866-b17d-8672d775fc4a","Type":"ContainerDied","Data":"a8c12da6975a29ad7545bec1427b4689b37026be94b62c4ac2af49f3f2e02053"} Oct 29 23:21:27.287358 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.286917 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5" event={"ID":"4cc22e58-ddce-4866-b17d-8672d775fc4a","Type":"ContainerDied","Data":"55e2c8a8f6588fa936ff10c780ce3bf92dfd70938f1d81415877fd38145db331"} Oct 29 23:21:27.287358 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.286950 5057 scope.go:122] "RemoveContainer" containerID="a8c12da6975a29ad7545bec1427b4689b37026be94b62c4ac2af49f3f2e02053" Oct 29 23:21:27.300334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.300243 5057 scope.go:122] "RemoveContainer" containerID="a8c12da6975a29ad7545bec1427b4689b37026be94b62c4ac2af49f3f2e02053" Oct 29 23:21:27.300571 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:21:27.300551 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c12da6975a29ad7545bec1427b4689b37026be94b62c4ac2af49f3f2e02053\": container with ID starting with a8c12da6975a29ad7545bec1427b4689b37026be94b62c4ac2af49f3f2e02053 not found: ID does not exist" containerID="a8c12da6975a29ad7545bec1427b4689b37026be94b62c4ac2af49f3f2e02053" Oct 29 23:21:27.300637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.300579 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c12da6975a29ad7545bec1427b4689b37026be94b62c4ac2af49f3f2e02053"} err="failed to get container status \"a8c12da6975a29ad7545bec1427b4689b37026be94b62c4ac2af49f3f2e02053\": rpc error: code = NotFound desc = could not find container \"a8c12da6975a29ad7545bec1427b4689b37026be94b62c4ac2af49f3f2e02053\": container with ID starting with a8c12da6975a29ad7545bec1427b4689b37026be94b62c4ac2af49f3f2e02053 not found: ID does not exist" Oct 29 23:21:27.353173 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.353138 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5"] Oct 29 23:21:27.368568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.368534 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-nblw5"] Oct 29 23:21:27.483323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:27.483288 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cc22e58-ddce-4866-b17d-8672d775fc4a" path="/var/lib/kubelet/pods/4cc22e58-ddce-4866-b17d-8672d775fc4a/volumes" Oct 29 23:21:32.512219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.510993 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r"] Oct 29 23:21:32.531221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.521321 5057 cpu_manager.go:414] "containerMap: removing container" podUID="fa2b610a-d82f-47f8-93cd-736a51e1d52e" containerName="istio-proxy" Oct 29 23:21:32.531221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.521352 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2b610a-d82f-47f8-93cd-736a51e1d52e" containerName="istio-proxy" Oct 29 23:21:32.531221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.521375 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4cc22e58-ddce-4866-b17d-8672d775fc4a" containerName="istio-proxy" Oct 29 23:21:32.531221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.521382 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc22e58-ddce-4866-b17d-8672d775fc4a" containerName="istio-proxy" Oct 29 23:21:32.531221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.521431 5057 cpu_manager.go:414] "containerMap: removing container" podUID="36f8ebdf-e0d2-4594-8d4c-9a177233e48a" containerName="istio-proxy" Oct 29 23:21:32.531221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.521439 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f8ebdf-e0d2-4594-8d4c-9a177233e48a" containerName="istio-proxy" Oct 29 23:21:32.531221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.521481 5057 cpu_manager.go:414] "containerMap: removing container" podUID="fa2b610a-d82f-47f8-93cd-736a51e1d52e" containerName="istio-proxy" Oct 29 23:21:32.531221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.521488 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2b610a-d82f-47f8-93cd-736a51e1d52e" containerName="istio-proxy" Oct 29 23:21:32.531221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.521786 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2b610a-d82f-47f8-93cd-736a51e1d52e" containerName="istio-proxy" Oct 29 23:21:32.531221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.521804 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc22e58-ddce-4866-b17d-8672d775fc4a" containerName="istio-proxy" Oct 29 23:21:32.531221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.521819 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f8ebdf-e0d2-4594-8d4c-9a177233e48a" containerName="istio-proxy" Oct 29 23:21:32.531221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.521832 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2b610a-d82f-47f8-93cd-736a51e1d52e" containerName="istio-proxy" Oct 29 23:21:32.535033 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.535004 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.628213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.626456 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-workload-certs\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.628213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.626519 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-istio-envoy\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.628213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.626707 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44008edf-7b81-4504-adb7-132388744c6e-istio-podinfo\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.628213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.626787 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54pdq\" (UniqueName: \"kubernetes.io/projected/44008edf-7b81-4504-adb7-132388744c6e-kube-api-access-54pdq\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.628213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.626847 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44008edf-7b81-4504-adb7-132388744c6e-istio-token\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.628213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.626890 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44008edf-7b81-4504-adb7-132388744c6e-istiod-ca-cert\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.628213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.627007 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-credential-socket\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.628213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.627055 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-workload-socket\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.628213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.627115 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-istio-data\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.830397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.801484 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-credential-socket\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.830397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.801555 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-workload-socket\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.830397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.801619 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-istio-data\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.830397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.801676 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-workload-certs\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.830397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.801716 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-istio-envoy\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.830397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.801876 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44008edf-7b81-4504-adb7-132388744c6e-istio-podinfo\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.830397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.801963 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54pdq\" (UniqueName: \"kubernetes.io/projected/44008edf-7b81-4504-adb7-132388744c6e-kube-api-access-54pdq\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.830397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.802027 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44008edf-7b81-4504-adb7-132388744c6e-istio-token\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.830397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.802071 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44008edf-7b81-4504-adb7-132388744c6e-istiod-ca-cert\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.830397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.803140 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44008edf-7b81-4504-adb7-132388744c6e-istiod-ca-cert\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.830397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.821794 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-istio-data\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.830397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.822081 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-workload-socket\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.849298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.849259 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-workload-certs\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.864933 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.851226 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-credential-socket\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.880429 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.880393 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r"] Oct 29 23:21:32.880923 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.880897 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a84df8-openshift-default-dockercfg-6tfrv\"" Oct 29 23:21:32.891879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.891815 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54pdq\" (UniqueName: \"kubernetes.io/projected/44008edf-7b81-4504-adb7-132388744c6e-kube-api-access-54pdq\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.895109 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.895055 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44008edf-7b81-4504-adb7-132388744c6e-istio-token\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.898656 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.898596 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-istio-envoy\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.901688 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.901429 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44008edf-7b81-4504-adb7-132388744c6e-istio-podinfo\") pod \"e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:32.939860 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:32.939835 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:33.341347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:33.341312 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r"] Oct 29 23:21:33.342004 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:21:33.341969 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44008edf_7b81_4504_adb7_132388744c6e.slice/crio-ea3fffd09860dda1343ee4b5a3ddc88bee53c058a4170f044d17c9f06745a2ff WatchSource:0}: Error finding container ea3fffd09860dda1343ee4b5a3ddc88bee53c058a4170f044d17c9f06745a2ff: Status 404 returned error can't find the container with id ea3fffd09860dda1343ee4b5a3ddc88bee53c058a4170f044d17c9f06745a2ff Oct 29 23:21:33.345925 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:33.345884 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:21:33.346061 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:33.345976 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:21:33.346061 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:33.346017 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:21:34.043211 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:21:34.034069 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:21:34.358135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:34.358032 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" event={"ID":"44008edf-7b81-4504-adb7-132388744c6e","Type":"ContainerStarted","Data":"286faffdb87fcb34fbf9536fc0e64fb232b29f1119fc5baed68bdd620efdf3f6"} Oct 29 23:21:34.358506 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:34.358480 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" event={"ID":"44008edf-7b81-4504-adb7-132388744c6e","Type":"ContainerStarted","Data":"ea3fffd09860dda1343ee4b5a3ddc88bee53c058a4170f044d17c9f06745a2ff"} Oct 29 23:21:34.439971 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:34.439922 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" podStartSLOduration=2.439908155 podStartE2EDuration="2.439908155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:34.422665496 +0000 UTC m=+2341.074164023" watchObservedRunningTime="2026-10-29 23:21:34.439908155 +0000 UTC m=+2341.091406748" Oct 29 23:21:34.940447 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:34.940416 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:34.951153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:34.951128 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:35.451211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:35.445705 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:35.467707 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:35.462604 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:21:36.516224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:36.492285 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:21:39.178465 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.171526 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9"] Oct 29 23:21:39.189374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.189336 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9" Oct 29 23:21:39.198545 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.198511 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9"] Oct 29 23:21:39.206566 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.206531 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a84df8\"" Oct 29 23:21:39.220722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.220689 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a84df8-dockercfg-7gb9b\"" Oct 29 23:21:39.290630 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.290136 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9\" (UID: \"5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9" Oct 29 23:21:39.307653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.307615 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shz7d\" (UniqueName: \"kubernetes.io/projected/5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b-kube-api-access-shz7d\") pod \"payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9\" (UID: \"5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9" Oct 29 23:21:39.330213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.326365 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj"] Oct 29 23:21:39.427211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.424604 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shz7d\" (UniqueName: \"kubernetes.io/projected/5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b-kube-api-access-shz7d\") pod \"payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9\" (UID: \"5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9" Oct 29 23:21:39.427211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.424820 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9\" (UID: \"5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9" Oct 29 23:21:39.427211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.425802 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9\" (UID: \"5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9" Oct 29 23:21:39.551749 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.551668 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj" Oct 29 23:21:39.697256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.697162 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shz7d\" (UniqueName: \"kubernetes.io/projected/5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b-kube-api-access-shz7d\") pod \"payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9\" (UID: \"5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9" Oct 29 23:21:39.712820 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.712791 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj"] Oct 29 23:21:39.794992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.794959 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm7wq\" (UniqueName: \"kubernetes.io/projected/153fed7b-8844-4d52-a5ad-0b6d8eda52e7-kube-api-access-gm7wq\") pod \"payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj\" (UID: \"153fed7b-8844-4d52-a5ad-0b6d8eda52e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj" Oct 29 23:21:39.795280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.795259 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153fed7b-8844-4d52-a5ad-0b6d8eda52e7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj\" (UID: \"153fed7b-8844-4d52-a5ad-0b6d8eda52e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj" Oct 29 23:21:39.811497 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.811424 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9" Oct 29 23:21:39.898332 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.897148 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm7wq\" (UniqueName: \"kubernetes.io/projected/153fed7b-8844-4d52-a5ad-0b6d8eda52e7-kube-api-access-gm7wq\") pod \"payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj\" (UID: \"153fed7b-8844-4d52-a5ad-0b6d8eda52e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj" Oct 29 23:21:39.898332 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.897380 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153fed7b-8844-4d52-a5ad-0b6d8eda52e7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj\" (UID: \"153fed7b-8844-4d52-a5ad-0b6d8eda52e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj" Oct 29 23:21:39.898332 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.898026 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153fed7b-8844-4d52-a5ad-0b6d8eda52e7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj\" (UID: \"153fed7b-8844-4d52-a5ad-0b6d8eda52e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj" Oct 29 23:21:39.994806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.985884 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm7wq\" (UniqueName: \"kubernetes.io/projected/153fed7b-8844-4d52-a5ad-0b6d8eda52e7-kube-api-access-gm7wq\") pod \"payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj\" (UID: \"153fed7b-8844-4d52-a5ad-0b6d8eda52e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj" Oct 29 23:21:39.996467 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:39.996150 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj" Oct 29 23:21:40.308420 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:40.302751 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9"] Oct 29 23:21:40.313214 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:21:40.313151 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7ac761_6de5_4d8a_97a5_f8f0e5cbfc9b.slice/crio-5db39611224dbfa8e4e337209d97e5a4cd9aa7ed7291d5923cbabf0d67704167 WatchSource:0}: Error finding container 5db39611224dbfa8e4e337209d97e5a4cd9aa7ed7291d5923cbabf0d67704167: Status 404 returned error can't find the container with id 5db39611224dbfa8e4e337209d97e5a4cd9aa7ed7291d5923cbabf0d67704167 Oct 29 23:21:40.611224 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:21:40.611174 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153fed7b_8844_4d52_a5ad_0b6d8eda52e7.slice/crio-5cd02d375d9e555072f7a40a271a728c090f7b73d7988c6fd8a1fd2cb41e9377 WatchSource:0}: Error finding container 5cd02d375d9e555072f7a40a271a728c090f7b73d7988c6fd8a1fd2cb41e9377: Status 404 returned error can't find the container with id 5cd02d375d9e555072f7a40a271a728c090f7b73d7988c6fd8a1fd2cb41e9377 Oct 29 23:21:40.611347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:40.611305 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj"] Oct 29 23:21:40.717932 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:40.717896 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9" event={"ID":"5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b","Type":"ContainerStarted","Data":"9c8f22e461d7d5785f7faa0e92c98a58940a94c46624e54bc047b508bb089055"} Oct 29 23:21:40.718110 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:40.717953 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9" event={"ID":"5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b","Type":"ContainerStarted","Data":"5db39611224dbfa8e4e337209d97e5a4cd9aa7ed7291d5923cbabf0d67704167"} Oct 29 23:21:40.718110 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:40.717994 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9" Oct 29 23:21:40.719515 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:40.719486 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj" event={"ID":"153fed7b-8844-4d52-a5ad-0b6d8eda52e7","Type":"ContainerStarted","Data":"5cd02d375d9e555072f7a40a271a728c090f7b73d7988c6fd8a1fd2cb41e9377"} Oct 29 23:21:40.774629 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:40.774571 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9" podStartSLOduration=1.774545379 podStartE2EDuration="1.774545379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:40.774272448 +0000 UTC m=+2347.425770979" watchObservedRunningTime="2026-10-29 23:21:40.774545379 +0000 UTC m=+2347.426043912" Oct 29 23:21:40.820320 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:40.820286 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7881-6dab6a"] Oct 29 23:21:40.856791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:40.856338 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-6dab6a" Oct 29 23:21:40.856791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:40.856424 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7881-6dab6a"] Oct 29 23:21:40.873475 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:40.873449 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xdwj4\"" Oct 29 23:21:40.962207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:40.952083 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6pzd\" (UniqueName: \"kubernetes.io/projected/db8a8450-6b1e-4f26-8e07-c24114b1c0a7-kube-api-access-m6pzd\") pod \"test-curl-7881-6dab6a\" (UID: \"db8a8450-6b1e-4f26-8e07-c24114b1c0a7\") " pod="opendatahub/test-curl-7881-6dab6a" Oct 29 23:21:41.053320 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:41.053288 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6pzd\" (UniqueName: \"kubernetes.io/projected/db8a8450-6b1e-4f26-8e07-c24114b1c0a7-kube-api-access-m6pzd\") pod \"test-curl-7881-6dab6a\" (UID: \"db8a8450-6b1e-4f26-8e07-c24114b1c0a7\") " pod="opendatahub/test-curl-7881-6dab6a" Oct 29 23:21:41.081131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:41.081104 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6pzd\" (UniqueName: \"kubernetes.io/projected/db8a8450-6b1e-4f26-8e07-c24114b1c0a7-kube-api-access-m6pzd\") pod \"test-curl-7881-6dab6a\" (UID: \"db8a8450-6b1e-4f26-8e07-c24114b1c0a7\") " pod="opendatahub/test-curl-7881-6dab6a" Oct 29 23:21:41.176137 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:41.176103 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-6dab6a" Oct 29 23:21:41.557266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:41.555252 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7881-6dab6a"] Oct 29 23:21:41.562933 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:21:41.562658 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb8a8450_6b1e_4f26_8e07_c24114b1c0a7.slice/crio-6f1c3c18ad2adcae417e561243259e532018357a0352d0117cd8b6358fa758da WatchSource:0}: Error finding container 6f1c3c18ad2adcae417e561243259e532018357a0352d0117cd8b6358fa758da: Status 404 returned error can't find the container with id 6f1c3c18ad2adcae417e561243259e532018357a0352d0117cd8b6358fa758da Oct 29 23:21:41.762209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:41.761412 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj" event={"ID":"153fed7b-8844-4d52-a5ad-0b6d8eda52e7","Type":"ContainerStarted","Data":"ced5fbebedd978b16c23844f46cf1105fd2c4249a979349af808685c0e2c6b76"} Oct 29 23:21:41.762971 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:41.762721 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj" Oct 29 23:21:41.766434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:41.766408 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7881-6dab6a" event={"ID":"db8a8450-6b1e-4f26-8e07-c24114b1c0a7","Type":"ContainerStarted","Data":"4eeee06021b8d2f06741ce3f35d3641815fe2a009d90f2c867ad916419f42ca6"} Oct 29 23:21:41.766629 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:41.766610 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7881-6dab6a" event={"ID":"db8a8450-6b1e-4f26-8e07-c24114b1c0a7","Type":"ContainerStarted","Data":"6f1c3c18ad2adcae417e561243259e532018357a0352d0117cd8b6358fa758da"} Oct 29 23:21:41.872371 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:41.872236 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7881-6dab6a" podStartSLOduration=1.8722167889999999 podStartE2EDuration="1.872216789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:41.871507382 +0000 UTC m=+2348.523005917" watchObservedRunningTime="2026-10-29 23:21:41.872216789 +0000 UTC m=+2348.523715321" Oct 29 23:21:41.873168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:41.873124 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj" podStartSLOduration=2.873109118 podStartE2EDuration="2.873109118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:41.819902363 +0000 UTC m=+2348.471400893" watchObservedRunningTime="2026-10-29 23:21:41.873109118 +0000 UTC m=+2348.524607651" Oct 29 23:21:42.757182 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:42.757146 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7881-6dab6a"] Oct 29 23:21:42.764708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:42.764674 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7881-6dab6a"] Oct 29 23:21:43.787421 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:43.787356 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7881-6dab6a" podUID="db8a8450-6b1e-4f26-8e07-c24114b1c0a7" containerName="test-curl-7881-6dab6a" containerID="cri-o://4eeee06021b8d2f06741ce3f35d3641815fe2a009d90f2c867ad916419f42ca6" gracePeriod=2 Oct 29 23:21:46.114285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:46.114252 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-6dab6a" Oct 29 23:21:46.220838 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:46.220807 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db8a8450-6b1e-4f26-8e07-c24114b1c0a7-kube-api-access-m6pzd\" (UniqueName: \"kubernetes.io/projected/db8a8450-6b1e-4f26-8e07-c24114b1c0a7-kube-api-access-m6pzd\") pod \"db8a8450-6b1e-4f26-8e07-c24114b1c0a7\" (UID: \"db8a8450-6b1e-4f26-8e07-c24114b1c0a7\") " Oct 29 23:21:46.223540 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:46.223505 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8a8450-6b1e-4f26-8e07-c24114b1c0a7-kube-api-access-m6pzd" pod "db8a8450-6b1e-4f26-8e07-c24114b1c0a7" (UID: "db8a8450-6b1e-4f26-8e07-c24114b1c0a7"). InnerVolumeSpecName "kube-api-access-m6pzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:46.322886 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:46.322806 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6pzd\" (UniqueName: \"kubernetes.io/projected/db8a8450-6b1e-4f26-8e07-c24114b1c0a7-kube-api-access-m6pzd\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:46.812168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:46.812087 5057 generic.go:356] "Generic (PLEG): container finished" podID="5c575af2-2a8a-4e83-adcc-7d6578c53235" containerID="c0db97b4b8cac18eb5f7eca8e2f9046c924cd55e37f6ee5d7299ba0c009eb3d9" exitCode=0 Oct 29 23:21:46.812394 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:46.812159 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" event={"ID":"5c575af2-2a8a-4e83-adcc-7d6578c53235","Type":"ContainerDied","Data":"c0db97b4b8cac18eb5f7eca8e2f9046c924cd55e37f6ee5d7299ba0c009eb3d9"} Oct 29 23:21:46.812709 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:46.812692 5057 scope.go:122] "RemoveContainer" containerID="c0db97b4b8cac18eb5f7eca8e2f9046c924cd55e37f6ee5d7299ba0c009eb3d9" Oct 29 23:21:46.813841 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:46.813645 5057 generic.go:356] "Generic (PLEG): container finished" podID="db8a8450-6b1e-4f26-8e07-c24114b1c0a7" containerID="4eeee06021b8d2f06741ce3f35d3641815fe2a009d90f2c867ad916419f42ca6" exitCode=137 Oct 29 23:21:46.813841 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:46.813704 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-6dab6a" Oct 29 23:21:46.813841 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:46.813745 5057 scope.go:122] "RemoveContainer" containerID="4eeee06021b8d2f06741ce3f35d3641815fe2a009d90f2c867ad916419f42ca6" Oct 29 23:21:46.814826 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:46.814797 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:21:46.814888 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:46.814874 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:21:46.814934 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:46.814912 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:21:46.826333 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:46.826313 5057 scope.go:122] "RemoveContainer" containerID="4eeee06021b8d2f06741ce3f35d3641815fe2a009d90f2c867ad916419f42ca6" Oct 29 23:21:46.826601 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:21:46.826582 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eeee06021b8d2f06741ce3f35d3641815fe2a009d90f2c867ad916419f42ca6\": container with ID starting with 4eeee06021b8d2f06741ce3f35d3641815fe2a009d90f2c867ad916419f42ca6 not found: ID does not exist" containerID="4eeee06021b8d2f06741ce3f35d3641815fe2a009d90f2c867ad916419f42ca6" Oct 29 23:21:46.826675 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:46.826606 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eeee06021b8d2f06741ce3f35d3641815fe2a009d90f2c867ad916419f42ca6"} err="failed to get container status \"4eeee06021b8d2f06741ce3f35d3641815fe2a009d90f2c867ad916419f42ca6\": rpc error: code = NotFound desc = could not find container \"4eeee06021b8d2f06741ce3f35d3641815fe2a009d90f2c867ad916419f42ca6\": container with ID starting with 4eeee06021b8d2f06741ce3f35d3641815fe2a009d90f2c867ad916419f42ca6 not found: ID does not exist" Oct 29 23:21:46.892543 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:21:46.892514 5057 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7881-6dab6a\" is forbidden: User \"system:node:ip-10-0-135-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-80.ec2.internal' and this object" podUID="db8a8450-6b1e-4f26-8e07-c24114b1c0a7" pod="opendatahub/test-curl-7881-6dab6a" Oct 29 23:21:47.482480 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:47.482449 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db8a8450-6b1e-4f26-8e07-c24114b1c0a7" path="/var/lib/kubelet/pods/db8a8450-6b1e-4f26-8e07-c24114b1c0a7/volumes" Oct 29 23:21:47.861267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:47.861132 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" event={"ID":"5c575af2-2a8a-4e83-adcc-7d6578c53235","Type":"ContainerStarted","Data":"cceaaf0130b1dac747c78237db3470ead33ef2647084c4297051d73fe08c043e"} Oct 29 23:21:48.392615 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:48.392582 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:21:48.393589 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:48.393553 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 29 23:21:48.393737 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:48.393611 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" podUID="5c575af2-2a8a-4e83-adcc-7d6578c53235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 29 23:21:49.104992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:49.104956 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:21:49.392025 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:49.391943 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 29 23:21:49.392025 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:49.392007 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" podUID="5c575af2-2a8a-4e83-adcc-7d6578c53235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 29 23:21:50.392861 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:50.392824 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 29 23:21:50.393277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:50.392888 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" podUID="5c575af2-2a8a-4e83-adcc-7d6578c53235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 29 23:21:51.392932 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:51.392899 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 29 23:21:51.393395 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:51.392959 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" podUID="5c575af2-2a8a-4e83-adcc-7d6578c53235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 29 23:21:51.769993 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:51.769962 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9" Oct 29 23:21:52.392060 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:52.392031 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 29 23:21:52.392241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:52.392090 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" podUID="5c575af2-2a8a-4e83-adcc-7d6578c53235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 29 23:21:52.574615 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:52.574579 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w"] Oct 29 23:21:52.575067 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:52.574787 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" podUID="11ebf457-f868-43b1-890e-5ccfb79934b4" containerName="istio-proxy" containerID="cri-o://8e32623555ab55dff298e7421c1d11a529c7ce5f4039553cf8e312cd8ef23798" gracePeriod=30 Oct 29 23:21:53.396917 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:53.396876 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:21:53.398343 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:53.398321 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:21:53.790865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:53.790838 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj" Oct 29 23:21:57.886985 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:57.886957 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:21:58.016965 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.016926 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-envoy\") pod \"11ebf457-f868-43b1-890e-5ccfb79934b4\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " Oct 29 23:21:58.017314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.017287 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-credential-socket\") pod \"11ebf457-f868-43b1-890e-5ccfb79934b4\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " Oct 29 23:21:58.017608 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.017589 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-podinfo\") pod \"11ebf457-f868-43b1-890e-5ccfb79934b4\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " Oct 29 23:21:58.017898 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.017613 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-credential-socket" pod "11ebf457-f868-43b1-890e-5ccfb79934b4" (UID: "11ebf457-f868-43b1-890e-5ccfb79934b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:58.017898 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.017885 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-workload-certs\") pod \"11ebf457-f868-43b1-890e-5ccfb79934b4\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " Oct 29 23:21:58.018098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.017953 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-workload-socket\") pod \"11ebf457-f868-43b1-890e-5ccfb79934b4\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " Oct 29 23:21:58.018098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.018000 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-data\") pod \"11ebf457-f868-43b1-890e-5ccfb79934b4\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " Oct 29 23:21:58.018098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.018033 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ebf457-f868-43b1-890e-5ccfb79934b4-kube-api-access-pqwnz\" (UniqueName: \"kubernetes.io/projected/11ebf457-f868-43b1-890e-5ccfb79934b4-kube-api-access-pqwnz\") pod \"11ebf457-f868-43b1-890e-5ccfb79934b4\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " Oct 29 23:21:58.018334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.018100 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-token\" (UniqueName: \"kubernetes.io/projected/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-token\") pod \"11ebf457-f868-43b1-890e-5ccfb79934b4\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " Oct 29 23:21:58.018334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.018263 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11ebf457-f868-43b1-890e-5ccfb79934b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ebf457-f868-43b1-890e-5ccfb79934b4-istiod-ca-cert\") pod \"11ebf457-f868-43b1-890e-5ccfb79934b4\" (UID: \"11ebf457-f868-43b1-890e-5ccfb79934b4\") " Oct 29 23:21:58.018334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.018263 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-workload-certs" pod "11ebf457-f868-43b1-890e-5ccfb79934b4" (UID: "11ebf457-f868-43b1-890e-5ccfb79934b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:58.018519 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.018497 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-data" pod "11ebf457-f868-43b1-890e-5ccfb79934b4" (UID: "11ebf457-f868-43b1-890e-5ccfb79934b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:58.018742 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.018721 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-workload-socket" pod "11ebf457-f868-43b1-890e-5ccfb79934b4" (UID: "11ebf457-f868-43b1-890e-5ccfb79934b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:58.019179 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.019139 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:58.019326 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.019310 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:58.019438 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.019424 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:58.019541 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.019526 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:58.019619 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.019433 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ebf457-f868-43b1-890e-5ccfb79934b4-istiod-ca-cert" pod "11ebf457-f868-43b1-890e-5ccfb79934b4" (UID: "11ebf457-f868-43b1-890e-5ccfb79934b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:21:58.019803 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.019776 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-envoy" pod "11ebf457-f868-43b1-890e-5ccfb79934b4" (UID: "11ebf457-f868-43b1-890e-5ccfb79934b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:58.021315 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.021279 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-token" pod "11ebf457-f868-43b1-890e-5ccfb79934b4" (UID: "11ebf457-f868-43b1-890e-5ccfb79934b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:58.021480 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.021453 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-podinfo" pod "11ebf457-f868-43b1-890e-5ccfb79934b4" (UID: "11ebf457-f868-43b1-890e-5ccfb79934b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:21:58.021776 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.021746 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ebf457-f868-43b1-890e-5ccfb79934b4-kube-api-access-pqwnz" pod "11ebf457-f868-43b1-890e-5ccfb79934b4" (UID: "11ebf457-f868-43b1-890e-5ccfb79934b4"). InnerVolumeSpecName "kube-api-access-pqwnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:58.035980 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.035952 5057 generic.go:356] "Generic (PLEG): container finished" podID="11ebf457-f868-43b1-890e-5ccfb79934b4" containerID="8e32623555ab55dff298e7421c1d11a529c7ce5f4039553cf8e312cd8ef23798" exitCode=0 Oct 29 23:21:58.036114 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.036000 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" event={"ID":"11ebf457-f868-43b1-890e-5ccfb79934b4","Type":"ContainerDied","Data":"8e32623555ab55dff298e7421c1d11a529c7ce5f4039553cf8e312cd8ef23798"} Oct 29 23:21:58.036114 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.036021 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" Oct 29 23:21:58.036114 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.036048 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w" event={"ID":"11ebf457-f868-43b1-890e-5ccfb79934b4","Type":"ContainerDied","Data":"5a432c2f8e30566e3b49c7d8d8cc6ee9e8d97cd02d3e2c246e79e088e48c184d"} Oct 29 23:21:58.038723 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.036080 5057 scope.go:122] "RemoveContainer" containerID="8e32623555ab55dff298e7421c1d11a529c7ce5f4039553cf8e312cd8ef23798" Oct 29 23:21:58.062343 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.062322 5057 scope.go:122] "RemoveContainer" containerID="8e32623555ab55dff298e7421c1d11a529c7ce5f4039553cf8e312cd8ef23798" Oct 29 23:21:58.062656 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:21:58.062637 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e32623555ab55dff298e7421c1d11a529c7ce5f4039553cf8e312cd8ef23798\": container with ID starting with 8e32623555ab55dff298e7421c1d11a529c7ce5f4039553cf8e312cd8ef23798 not found: ID does not exist" containerID="8e32623555ab55dff298e7421c1d11a529c7ce5f4039553cf8e312cd8ef23798" Oct 29 23:21:58.062719 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.062664 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e32623555ab55dff298e7421c1d11a529c7ce5f4039553cf8e312cd8ef23798"} err="failed to get container status \"8e32623555ab55dff298e7421c1d11a529c7ce5f4039553cf8e312cd8ef23798\": rpc error: code = NotFound desc = could not find container \"8e32623555ab55dff298e7421c1d11a529c7ce5f4039553cf8e312cd8ef23798\": container with ID starting with 8e32623555ab55dff298e7421c1d11a529c7ce5f4039553cf8e312cd8ef23798 not found: ID does not exist" Oct 29 23:21:58.103403 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.103299 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w"] Oct 29 23:21:58.107101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.107072 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pmf8w"] Oct 29 23:21:58.121029 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.121002 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:58.121029 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.121029 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ebf457-f868-43b1-890e-5ccfb79934b4-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:58.121288 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.121042 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:58.121288 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.121054 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ebf457-f868-43b1-890e-5ccfb79934b4-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:58.121288 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:58.121065 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqwnz\" (UniqueName: \"kubernetes.io/projected/11ebf457-f868-43b1-890e-5ccfb79934b4-kube-api-access-pqwnz\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:21:59.497209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:21:59.495021 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11ebf457-f868-43b1-890e-5ccfb79934b4" path="/var/lib/kubelet/pods/11ebf457-f868-43b1-890e-5ccfb79934b4/volumes" Oct 29 23:22:03.916670 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:22:03.916632 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:22:04.592693 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:04.592654 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg"] Oct 29 23:22:06.675306 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:06.675264 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8"] Oct 29 23:22:06.676906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:06.676525 5057 cpu_manager.go:414] "containerMap: removing container" podUID="db8a8450-6b1e-4f26-8e07-c24114b1c0a7" containerName="test-curl-7881-6dab6a" Oct 29 23:22:06.676906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:06.676550 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db8a8450-6b1e-4f26-8e07-c24114b1c0a7" containerName="test-curl-7881-6dab6a" Oct 29 23:22:06.676906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:06.676603 5057 cpu_manager.go:414] "containerMap: removing container" podUID="11ebf457-f868-43b1-890e-5ccfb79934b4" containerName="istio-proxy" Oct 29 23:22:06.676906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:06.676611 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ebf457-f868-43b1-890e-5ccfb79934b4" containerName="istio-proxy" Oct 29 23:22:06.676906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:06.676800 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ebf457-f868-43b1-890e-5ccfb79934b4" containerName="istio-proxy" Oct 29 23:22:06.676906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:06.676825 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="db8a8450-6b1e-4f26-8e07-c24114b1c0a7" containerName="test-curl-7881-6dab6a" Oct 29 23:22:06.680515 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:06.680488 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8"] Oct 29 23:22:06.680632 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:06.680583 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:06.704283 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:06.704252 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-6f162a-openshift-default-dockercfg-fjf9s\"" Oct 29 23:22:06.950260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:06.946537 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-credential-socket\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:06.976303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:06.976259 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/271ab84e-ada3-4de0-8cbb-b25424172421-istiod-ca-cert\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:06.976689 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:06.976665 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcspz\" (UniqueName: \"kubernetes.io/projected/271ab84e-ada3-4de0-8cbb-b25424172421-kube-api-access-dcspz\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:06.981220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:06.976938 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/271ab84e-ada3-4de0-8cbb-b25424172421-istio-podinfo\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:06.988215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:06.981452 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-istio-data\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:06.988471 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:06.988442 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/271ab84e-ada3-4de0-8cbb-b25424172421-istio-token\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:06.993528 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:06.992432 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-istio-envoy\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:06.993528 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:06.992491 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-workload-socket\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:06.993528 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:06.992541 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-workload-certs\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:07.107207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.102255 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-workload-certs\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:07.107207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.102397 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-credential-socket\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:07.107207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.102423 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/271ab84e-ada3-4de0-8cbb-b25424172421-istiod-ca-cert\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:07.107207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.102525 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcspz\" (UniqueName: \"kubernetes.io/projected/271ab84e-ada3-4de0-8cbb-b25424172421-kube-api-access-dcspz\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:07.107207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.102620 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/271ab84e-ada3-4de0-8cbb-b25424172421-istio-podinfo\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:07.107207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.102664 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-istio-data\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:07.107207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.102699 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/271ab84e-ada3-4de0-8cbb-b25424172421-istio-token\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:07.107207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.102875 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-istio-envoy\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:07.107207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.102912 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-workload-socket\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:07.107207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.103437 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-workload-socket\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:07.107207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.103689 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-workload-certs\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:07.107207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.103916 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-credential-socket\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:07.107207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.105612 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-istio-data\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:07.110919 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.104626 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/271ab84e-ada3-4de0-8cbb-b25424172421-istiod-ca-cert\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:07.113373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.113315 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-istio-envoy\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:07.137462 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.137427 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/271ab84e-ada3-4de0-8cbb-b25424172421-istio-podinfo\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:07.149347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.149304 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcspz\" (UniqueName: \"kubernetes.io/projected/271ab84e-ada3-4de0-8cbb-b25424172421-kube-api-access-dcspz\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:07.151170 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.151146 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/271ab84e-ada3-4de0-8cbb-b25424172421-istio-token\") pod \"e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:07.305234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.305150 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:07.636318 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:22:07.636289 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271ab84e_ada3_4de0_8cbb_b25424172421.slice/crio-c8203a73658e27b314d43ce7bd5e6a83b95b5c81abecb85eebcbf95f18fb8edc WatchSource:0}: Error finding container c8203a73658e27b314d43ce7bd5e6a83b95b5c81abecb85eebcbf95f18fb8edc: Status 404 returned error can't find the container with id c8203a73658e27b314d43ce7bd5e6a83b95b5c81abecb85eebcbf95f18fb8edc Oct 29 23:22:07.638001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.637972 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8"] Oct 29 23:22:07.639759 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.639718 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:22:07.639847 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.639805 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:22:07.639885 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:07.639848 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:22:08.155765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:08.155722 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" event={"ID":"271ab84e-ada3-4de0-8cbb-b25424172421","Type":"ContainerStarted","Data":"77b1cb97d7cc92f7183a56aca462767ede0342c308d7ae2382514b54e6e52bf1"} Oct 29 23:22:08.180851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:08.155829 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" event={"ID":"271ab84e-ada3-4de0-8cbb-b25424172421","Type":"ContainerStarted","Data":"c8203a73658e27b314d43ce7bd5e6a83b95b5c81abecb85eebcbf95f18fb8edc"} Oct 29 23:22:08.269248 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:08.260864 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" podStartSLOduration=2.260840318 podStartE2EDuration="2.260840318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:08.223147374 +0000 UTC m=+2374.874645906" watchObservedRunningTime="2026-10-29 23:22:08.260840318 +0000 UTC m=+2374.912338850" Oct 29 23:22:08.309209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:08.308578 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:08.321671 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:08.315697 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 29 23:22:08.321671 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:08.315772 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" podUID="271ab84e-ada3-4de0-8cbb-b25424172421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 29 23:22:09.310910 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:09.310879 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:10.172357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:10.172325 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:10.173362 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:10.173339 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:22:11.315456 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:11.315417 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg"] Oct 29 23:22:11.315897 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:11.315663 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" podUID="531e93a3-f8f2-4b85-966c-e1830958da9f" containerName="istio-proxy" containerID="cri-o://d4cb6aa44d50c4e8a1792b169137918cb6a3a9c992b54ac4abc25fcdaffb63a9" gracePeriod=30 Oct 29 23:22:12.502242 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:12.500420 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt"] Oct 29 23:22:12.975825 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:12.975792 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt"] Oct 29 23:22:12.976224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:12.976179 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.010323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.010290 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-c1e164-openshift-default-dockercfg-b6mtl\"" Oct 29 23:22:13.144410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.144374 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istiod-ca-cert\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.144678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.144592 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-token\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.144795 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.144774 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-envoy\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.144944 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.144819 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-podinfo\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.145023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.144975 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-workload-socket\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.145081 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.145026 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhbk6\" (UniqueName: \"kubernetes.io/projected/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-kube-api-access-fhbk6\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.145131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.145094 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-data\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.145131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.145121 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-credential-socket\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.145249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.145224 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-workload-certs\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.254903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.252583 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istiod-ca-cert\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.254903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.252728 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-token\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.254903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.252808 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-envoy\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.254903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.252844 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-podinfo\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.254903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.252903 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-workload-socket\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.254903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.252934 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhbk6\" (UniqueName: \"kubernetes.io/projected/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-kube-api-access-fhbk6\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.254903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.252998 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-data\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.254903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.253021 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-credential-socket\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.254903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.253071 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-workload-certs\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.254903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.253697 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-workload-certs\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.254903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.253942 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-workload-socket\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.254903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.254586 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-data\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.254903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.254843 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-credential-socket\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.358210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.357717 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istiod-ca-cert\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.382853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.382819 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-podinfo\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.410220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.408698 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-token\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.442223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.440725 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-envoy\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.452214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.452167 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhbk6\" (UniqueName: \"kubernetes.io/projected/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-kube-api-access-fhbk6\") pod \"e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.599853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.599775 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:13.967606 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.967572 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt"] Oct 29 23:22:13.967737 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:22:13.967699 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fcd56c2_48d5_47cc_a17f_7709d410e2b1.slice/crio-d9193febdb7bfac76f9dcfc497d7f072beeb9767c6765a6363ebb05a454701d5 WatchSource:0}: Error finding container d9193febdb7bfac76f9dcfc497d7f072beeb9767c6765a6363ebb05a454701d5: Status 404 returned error can't find the container with id d9193febdb7bfac76f9dcfc497d7f072beeb9767c6765a6363ebb05a454701d5 Oct 29 23:22:13.970657 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.970621 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:22:13.970811 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.970709 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:22:13.970811 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:13.970744 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:22:14.273261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:14.273059 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" event={"ID":"1fcd56c2-48d5-47cc-a17f-7709d410e2b1","Type":"ContainerStarted","Data":"badcc38556eeb0eb1a8896d8adb3bf7cb58001749954cc94dd3fe73d36831845"} Oct 29 23:22:14.273261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:14.273132 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" event={"ID":"1fcd56c2-48d5-47cc-a17f-7709d410e2b1","Type":"ContainerStarted","Data":"d9193febdb7bfac76f9dcfc497d7f072beeb9767c6765a6363ebb05a454701d5"} Oct 29 23:22:14.339165 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:14.339082 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" podStartSLOduration=2.339064327 podStartE2EDuration="2.339064327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:14.338007755 +0000 UTC m=+2380.989506285" watchObservedRunningTime="2026-10-29 23:22:14.339064327 +0000 UTC m=+2380.990562861" Oct 29 23:22:14.600424 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:14.600348 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:15.057500 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:15.057460 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:22:15.057684 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:15.057532 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" podUID="531e93a3-f8f2-4b85-966c-e1830958da9f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:22:15.607120 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:15.607089 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:16.290582 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.290503 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:16.291568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.291545 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:22:16.642069 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.642034 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 23:22:16.839941 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.839901 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-workload-socket\") pod \"531e93a3-f8f2-4b85-966c-e1830958da9f\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " Oct 29 23:22:16.840267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.839957 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/531e93a3-f8f2-4b85-966c-e1830958da9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/531e93a3-f8f2-4b85-966c-e1830958da9f-istiod-ca-cert\") pod \"531e93a3-f8f2-4b85-966c-e1830958da9f\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " Oct 29 23:22:16.840267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.840006 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-token\" (UniqueName: \"kubernetes.io/projected/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-token\") pod \"531e93a3-f8f2-4b85-966c-e1830958da9f\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " Oct 29 23:22:16.840267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.840157 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-workload-certs\") pod \"531e93a3-f8f2-4b85-966c-e1830958da9f\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " Oct 29 23:22:16.840267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.840199 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-workload-socket" pod "531e93a3-f8f2-4b85-966c-e1830958da9f" (UID: "531e93a3-f8f2-4b85-966c-e1830958da9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:16.840267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.840224 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/531e93a3-f8f2-4b85-966c-e1830958da9f-kube-api-access-pngrj\" (UniqueName: \"kubernetes.io/projected/531e93a3-f8f2-4b85-966c-e1830958da9f-kube-api-access-pngrj\") pod \"531e93a3-f8f2-4b85-966c-e1830958da9f\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " Oct 29 23:22:16.840267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.840260 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-data\") pod \"531e93a3-f8f2-4b85-966c-e1830958da9f\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " Oct 29 23:22:16.840602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.840370 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-envoy\") pod \"531e93a3-f8f2-4b85-966c-e1830958da9f\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " Oct 29 23:22:16.840602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.840416 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-workload-certs" pod "531e93a3-f8f2-4b85-966c-e1830958da9f" (UID: "531e93a3-f8f2-4b85-966c-e1830958da9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:16.840602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.840449 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/531e93a3-f8f2-4b85-966c-e1830958da9f-istiod-ca-cert" pod "531e93a3-f8f2-4b85-966c-e1830958da9f" (UID: "531e93a3-f8f2-4b85-966c-e1830958da9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:22:16.840602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.840481 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-credential-socket\") pod \"531e93a3-f8f2-4b85-966c-e1830958da9f\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " Oct 29 23:22:16.840602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.840531 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-podinfo\") pod \"531e93a3-f8f2-4b85-966c-e1830958da9f\" (UID: \"531e93a3-f8f2-4b85-966c-e1830958da9f\") " Oct 29 23:22:16.840866 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.840603 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-data" pod "531e93a3-f8f2-4b85-966c-e1830958da9f" (UID: "531e93a3-f8f2-4b85-966c-e1830958da9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:16.840866 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.840738 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-credential-socket" pod "531e93a3-f8f2-4b85-966c-e1830958da9f" (UID: "531e93a3-f8f2-4b85-966c-e1830958da9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:16.841388 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.841361 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:22:16.841518 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.841389 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:22:16.841518 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.841403 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/531e93a3-f8f2-4b85-966c-e1830958da9f-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:22:16.841518 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.841420 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:22:16.841518 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.841433 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:22:16.843606 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.843583 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-envoy" pod "531e93a3-f8f2-4b85-966c-e1830958da9f" (UID: "531e93a3-f8f2-4b85-966c-e1830958da9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:16.843919 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.843895 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531e93a3-f8f2-4b85-966c-e1830958da9f-kube-api-access-pngrj" pod "531e93a3-f8f2-4b85-966c-e1830958da9f" (UID: "531e93a3-f8f2-4b85-966c-e1830958da9f"). InnerVolumeSpecName "kube-api-access-pngrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:22:16.845632 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.845608 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-token" pod "531e93a3-f8f2-4b85-966c-e1830958da9f" (UID: "531e93a3-f8f2-4b85-966c-e1830958da9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:22:16.846008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.845983 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-podinfo" pod "531e93a3-f8f2-4b85-966c-e1830958da9f" (UID: "531e93a3-f8f2-4b85-966c-e1830958da9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:22:16.943750 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.943713 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:22:16.943750 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.943750 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:22:16.944047 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.943764 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pngrj\" (UniqueName: \"kubernetes.io/projected/531e93a3-f8f2-4b85-966c-e1830958da9f-kube-api-access-pngrj\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:22:16.944047 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:16.943779 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/531e93a3-f8f2-4b85-966c-e1830958da9f-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:22:17.299894 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:17.299857 5057 generic.go:356] "Generic (PLEG): container finished" podID="531e93a3-f8f2-4b85-966c-e1830958da9f" containerID="d4cb6aa44d50c4e8a1792b169137918cb6a3a9c992b54ac4abc25fcdaffb63a9" exitCode=0 Oct 29 23:22:17.300231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:17.299937 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" Oct 29 23:22:17.300433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:17.299928 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" event={"ID":"531e93a3-f8f2-4b85-966c-e1830958da9f","Type":"ContainerDied","Data":"d4cb6aa44d50c4e8a1792b169137918cb6a3a9c992b54ac4abc25fcdaffb63a9"} Oct 29 23:22:17.300711 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:17.300398 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg" event={"ID":"531e93a3-f8f2-4b85-966c-e1830958da9f","Type":"ContainerDied","Data":"c888ff78a40f1e69723dff3fe0e63a7ac7049948d1eccd67bcbdbfe6197729e9"} Oct 29 23:22:17.300802 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:17.300477 5057 scope.go:122] "RemoveContainer" containerID="d4cb6aa44d50c4e8a1792b169137918cb6a3a9c992b54ac4abc25fcdaffb63a9" Oct 29 23:22:17.433253 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:17.419415 5057 scope.go:122] "RemoveContainer" containerID="d4cb6aa44d50c4e8a1792b169137918cb6a3a9c992b54ac4abc25fcdaffb63a9" Oct 29 23:22:17.445210 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:22:17.441397 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4cb6aa44d50c4e8a1792b169137918cb6a3a9c992b54ac4abc25fcdaffb63a9\": container with ID starting with d4cb6aa44d50c4e8a1792b169137918cb6a3a9c992b54ac4abc25fcdaffb63a9 not found: ID does not exist" containerID="d4cb6aa44d50c4e8a1792b169137918cb6a3a9c992b54ac4abc25fcdaffb63a9" Oct 29 23:22:17.445210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:17.441463 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4cb6aa44d50c4e8a1792b169137918cb6a3a9c992b54ac4abc25fcdaffb63a9"} err="failed to get container status \"d4cb6aa44d50c4e8a1792b169137918cb6a3a9c992b54ac4abc25fcdaffb63a9\": rpc error: code = NotFound desc = could not find container \"d4cb6aa44d50c4e8a1792b169137918cb6a3a9c992b54ac4abc25fcdaffb63a9\": container with ID starting with d4cb6aa44d50c4e8a1792b169137918cb6a3a9c992b54ac4abc25fcdaffb63a9 not found: ID does not exist" Oct 29 23:22:17.445210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:17.441537 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg"] Oct 29 23:22:17.473143 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:17.473010 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-6v2hg"] Oct 29 23:22:17.520758 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:17.520711 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="531e93a3-f8f2-4b85-966c-e1830958da9f" path="/var/lib/kubelet/pods/531e93a3-f8f2-4b85-966c-e1830958da9f/volumes" Oct 29 23:22:19.194631 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.194595 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7"] Oct 29 23:22:19.196759 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.196731 5057 cpu_manager.go:414] "containerMap: removing container" podUID="531e93a3-f8f2-4b85-966c-e1830958da9f" containerName="istio-proxy" Oct 29 23:22:19.196931 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.196913 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="531e93a3-f8f2-4b85-966c-e1830958da9f" containerName="istio-proxy" Oct 29 23:22:19.197261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.197241 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="531e93a3-f8f2-4b85-966c-e1830958da9f" containerName="istio-proxy" Oct 29 23:22:19.212211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.202141 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.241236 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.223138 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7"] Oct 29 23:22:19.241236 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.233076 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-e10975-openshift-default-dockercfg-rhgnk\"" Oct 29 23:22:19.315638 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.315600 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-credential-socket\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.316000 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.315972 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-podinfo\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.316357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.316335 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-envoy\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.316577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.316556 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jppz5\" (UniqueName: \"kubernetes.io/projected/b7893cca-8dc6-4395-bc25-452bcd512dcd-kube-api-access-jppz5\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.316760 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.316742 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-workload-certs\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.316947 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.316929 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-data\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.317144 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.317122 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-workload-socket\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.317288 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.317169 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-token\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.317355 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.317319 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7893cca-8dc6-4395-bc25-452bcd512dcd-istiod-ca-cert\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.431212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.428264 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-workload-socket\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.431212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.428323 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-token\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.431212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.428423 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7893cca-8dc6-4395-bc25-452bcd512dcd-istiod-ca-cert\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.431212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.428571 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-credential-socket\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.431212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.428630 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-podinfo\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.431212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.428753 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-envoy\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.431212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.428798 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jppz5\" (UniqueName: \"kubernetes.io/projected/b7893cca-8dc6-4395-bc25-452bcd512dcd-kube-api-access-jppz5\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.431212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.428823 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-workload-certs\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.431212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.428859 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-data\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.431212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.429362 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-data\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.431212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.429750 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-workload-socket\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.431212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.430697 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7893cca-8dc6-4395-bc25-452bcd512dcd-istiod-ca-cert\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.431212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.430936 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-credential-socket\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.447268 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.432112 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-workload-certs\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.487210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.484717 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-envoy\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.508209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.493503 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-podinfo\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.508209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.505824 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-token\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.517210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.508901 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jppz5\" (UniqueName: \"kubernetes.io/projected/b7893cca-8dc6-4395-bc25-452bcd512dcd-kube-api-access-jppz5\") pod \"e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.557212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.552688 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:19.652220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.641731 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" podUID="0014c879-9ec4-4197-984a-f10a20b6510e" containerName="main" containerID="cri-o://3f3b44a8f33f9f68d26b313ccbe640da7bbff70904f3d0b54a367e7aac23ab3e" gracePeriod=45 Oct 29 23:22:19.794434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:19.794331 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" podUID="0014c879-9ec4-4197-984a-f10a20b6510e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.226:8000/ready\": EOF" Oct 29 23:22:20.105308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.105275 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7"] Oct 29 23:22:20.106381 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:22:20.106350 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7893cca_8dc6_4395_bc25_452bcd512dcd.slice/crio-7bdffc0fcd5bf3e94ba171a26ad0431f47cf7ab7e2f5a2f6c60f9ca58ba3e569 WatchSource:0}: Error finding container 7bdffc0fcd5bf3e94ba171a26ad0431f47cf7ab7e2f5a2f6c60f9ca58ba3e569: Status 404 returned error can't find the container with id 7bdffc0fcd5bf3e94ba171a26ad0431f47cf7ab7e2f5a2f6c60f9ca58ba3e569 Oct 29 23:22:20.109779 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.109743 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:22:20.109887 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.109830 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:22:20.109887 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.109879 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:22:20.556215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.549579 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:22:20.595015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.594338 5057 generic.go:356] "Generic (PLEG): container finished" podID="0014c879-9ec4-4197-984a-f10a20b6510e" containerID="3f3b44a8f33f9f68d26b313ccbe640da7bbff70904f3d0b54a367e7aac23ab3e" exitCode=0 Oct 29 23:22:20.595015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.594421 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" event={"ID":"0014c879-9ec4-4197-984a-f10a20b6510e","Type":"ContainerDied","Data":"3f3b44a8f33f9f68d26b313ccbe640da7bbff70904f3d0b54a367e7aac23ab3e"} Oct 29 23:22:20.595015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.594470 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" event={"ID":"0014c879-9ec4-4197-984a-f10a20b6510e","Type":"ContainerDied","Data":"1141f5208030b70373e1a771a5d651998ac26e1cfefd1256802fd512cb038f09"} Oct 29 23:22:20.595015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.594492 5057 scope.go:122] "RemoveContainer" containerID="3f3b44a8f33f9f68d26b313ccbe640da7bbff70904f3d0b54a367e7aac23ab3e" Oct 29 23:22:20.601024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.600394 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" event={"ID":"b7893cca-8dc6-4395-bc25-452bcd512dcd","Type":"ContainerStarted","Data":"98c68d250dbc7aaad45f78afa67d857bb15b249353ccca9a8b33c33ac1f9aa66"} Oct 29 23:22:20.601024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.600440 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" event={"ID":"b7893cca-8dc6-4395-bc25-452bcd512dcd","Type":"ContainerStarted","Data":"7bdffc0fcd5bf3e94ba171a26ad0431f47cf7ab7e2f5a2f6c60f9ca58ba3e569"} Oct 29 23:22:20.612963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.612721 5057 scope.go:122] "RemoveContainer" containerID="3f3b44a8f33f9f68d26b313ccbe640da7bbff70904f3d0b54a367e7aac23ab3e" Oct 29 23:22:20.613108 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:22:20.613058 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f3b44a8f33f9f68d26b313ccbe640da7bbff70904f3d0b54a367e7aac23ab3e\": container with ID starting with 3f3b44a8f33f9f68d26b313ccbe640da7bbff70904f3d0b54a367e7aac23ab3e not found: ID does not exist" containerID="3f3b44a8f33f9f68d26b313ccbe640da7bbff70904f3d0b54a367e7aac23ab3e" Oct 29 23:22:20.613163 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.613122 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3b44a8f33f9f68d26b313ccbe640da7bbff70904f3d0b54a367e7aac23ab3e"} err="failed to get container status \"3f3b44a8f33f9f68d26b313ccbe640da7bbff70904f3d0b54a367e7aac23ab3e\": rpc error: code = NotFound desc = could not find container \"3f3b44a8f33f9f68d26b313ccbe640da7bbff70904f3d0b54a367e7aac23ab3e\": container with ID starting with 3f3b44a8f33f9f68d26b313ccbe640da7bbff70904f3d0b54a367e7aac23ab3e not found: ID does not exist" Oct 29 23:22:20.699484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.698903 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-tmp-dir\") pod \"0014c879-9ec4-4197-984a-f10a20b6510e\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " Oct 29 23:22:20.699484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.698946 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0014c879-9ec4-4197-984a-f10a20b6510e-tls-certs\" (UniqueName: \"kubernetes.io/secret/0014c879-9ec4-4197-984a-f10a20b6510e-tls-certs\") pod \"0014c879-9ec4-4197-984a-f10a20b6510e\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " Oct 29 23:22:20.699484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.698985 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-home\" (UniqueName: \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-home\") pod \"0014c879-9ec4-4197-984a-f10a20b6510e\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " Oct 29 23:22:20.699484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.699048 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-model-cache\") pod \"0014c879-9ec4-4197-984a-f10a20b6510e\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " Oct 29 23:22:20.699484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.699146 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-dshm\") pod \"0014c879-9ec4-4197-984a-f10a20b6510e\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " Oct 29 23:22:20.699484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.699177 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0014c879-9ec4-4197-984a-f10a20b6510e-kube-api-access-74ngh\" (UniqueName: \"kubernetes.io/projected/0014c879-9ec4-4197-984a-f10a20b6510e-kube-api-access-74ngh\") pod \"0014c879-9ec4-4197-984a-f10a20b6510e\" (UID: \"0014c879-9ec4-4197-984a-f10a20b6510e\") " Oct 29 23:22:20.699484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.699226 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-tmp-dir" pod "0014c879-9ec4-4197-984a-f10a20b6510e" (UID: "0014c879-9ec4-4197-984a-f10a20b6510e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:20.700428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.700345 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-tmp-dir\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:22:20.700561 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.700456 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-home" pod "0014c879-9ec4-4197-984a-f10a20b6510e" (UID: "0014c879-9ec4-4197-984a-f10a20b6510e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:20.700875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.700634 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-model-cache" pod "0014c879-9ec4-4197-984a-f10a20b6510e" (UID: "0014c879-9ec4-4197-984a-f10a20b6510e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:20.703656 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.703606 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" podStartSLOduration=1.703588416 podStartE2EDuration="1.703588416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:20.689743078 +0000 UTC m=+2387.341241606" watchObservedRunningTime="2026-10-29 23:22:20.703588416 +0000 UTC m=+2387.355086952" Oct 29 23:22:20.705252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.705218 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0014c879-9ec4-4197-984a-f10a20b6510e-tls-certs" pod "0014c879-9ec4-4197-984a-f10a20b6510e" (UID: "0014c879-9ec4-4197-984a-f10a20b6510e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:22:20.705607 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.705585 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-dshm" pod "0014c879-9ec4-4197-984a-f10a20b6510e" (UID: "0014c879-9ec4-4197-984a-f10a20b6510e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:20.705857 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.705832 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0014c879-9ec4-4197-984a-f10a20b6510e-kube-api-access-74ngh" pod "0014c879-9ec4-4197-984a-f10a20b6510e" (UID: "0014c879-9ec4-4197-984a-f10a20b6510e"). InnerVolumeSpecName "kube-api-access-74ngh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:22:20.839963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.838900 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-model-cache\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:22:20.839963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.838938 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-dshm\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:22:20.839963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.838955 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74ngh\" (UniqueName: \"kubernetes.io/projected/0014c879-9ec4-4197-984a-f10a20b6510e-kube-api-access-74ngh\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:22:20.839963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.838969 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0014c879-9ec4-4197-984a-f10a20b6510e-tls-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:22:20.839963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:20.838982 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0014c879-9ec4-4197-984a-f10a20b6510e-home\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:22:21.504305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:21.504269 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" podUID="0014c879-9ec4-4197-984a-f10a20b6510e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.226:8000/ready\": context deadline exceeded" Oct 29 23:22:21.504443 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:21.504348 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:22:21.556598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:21.556557 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:21.564248 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:21.564217 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:21.610659 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:21.610627 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg" Oct 29 23:22:21.611010 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:21.610982 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:21.612117 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:21.612094 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:22:21.673607 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:21.673573 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg"] Oct 29 23:22:21.680484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:21.680456 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4ad17e/auth-test-model-4ad17e-kserve-779764f6cc-bk4zg"] Oct 29 23:22:23.614211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:23.612996 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0014c879-9ec4-4197-984a-f10a20b6510e" path="/var/lib/kubelet/pods/0014c879-9ec4-4197-984a-f10a20b6510e/volumes" Oct 29 23:22:26.340212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:26.339940 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:22:33.914965 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:22:33.914924 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:22:34.973551 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:34.973520 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:22:34.991745 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:34.991720 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:22:59.042342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:59.042285 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:22:59.044473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:59.044437 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 23:22:59.164516 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:59.164474 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:22:59.489843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:22:59.488374 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 23:23:03.921295 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:23:03.921239 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:23:08.745543 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:08.745504 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd"] Oct 29 23:23:08.746076 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:08.745728 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" podUID="227efff8-cd76-476b-baa6-045e35ab4d4f" containerName="istio-proxy" containerID="cri-o://62661df807d64166aab9fa375ac64c909da5e669aef0ed4608277ba2c97dde20" gracePeriod=30 Oct 29 23:23:09.833913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:09.833879 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg"] Oct 29 23:23:09.835228 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:09.835175 5057 cpu_manager.go:414] "containerMap: removing container" podUID="0014c879-9ec4-4197-984a-f10a20b6510e" containerName="main" Oct 29 23:23:09.835228 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:09.835215 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0014c879-9ec4-4197-984a-f10a20b6510e" containerName="main" Oct 29 23:23:09.835436 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:09.835409 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="0014c879-9ec4-4197-984a-f10a20b6510e" containerName="main" Oct 29 23:23:09.839439 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:09.839415 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" Oct 29 23:23:09.864177 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:09.864146 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg"] Oct 29 23:23:09.872453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:09.872423 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-c1e164-serving-cert\"" Oct 29 23:23:09.883472 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:09.883437 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6585z\" (UniqueName: \"kubernetes.io/projected/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-kube-api-access-6585z\") pod \"maas-api-e2e-autores-c1e164-847ddc8596-4gnsg\" (UID: \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" Oct 29 23:23:09.883777 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:09.883742 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-metrics-certs\") pod \"maas-api-e2e-autores-c1e164-847ddc8596-4gnsg\" (UID: \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" Oct 29 23:23:09.883952 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:09.883937 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-maas-api-tls\") pod \"maas-api-e2e-autores-c1e164-847ddc8596-4gnsg\" (UID: \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" Oct 29 23:23:09.884101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:09.884088 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-openshift-service-ca\") pod \"maas-api-e2e-autores-c1e164-847ddc8596-4gnsg\" (UID: \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" Oct 29 23:23:09.985850 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:09.985812 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6585z\" (UniqueName: \"kubernetes.io/projected/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-kube-api-access-6585z\") pod \"maas-api-e2e-autores-c1e164-847ddc8596-4gnsg\" (UID: \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" Oct 29 23:23:09.986041 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:09.985911 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-metrics-certs\") pod \"maas-api-e2e-autores-c1e164-847ddc8596-4gnsg\" (UID: \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" Oct 29 23:23:09.986041 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:09.986007 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-maas-api-tls\") pod \"maas-api-e2e-autores-c1e164-847ddc8596-4gnsg\" (UID: \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" Oct 29 23:23:09.986154 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:09.986131 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-openshift-service-ca\") pod \"maas-api-e2e-autores-c1e164-847ddc8596-4gnsg\" (UID: \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" Oct 29 23:23:09.986721 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:23:09.986434 5057 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-serving-cert: secret "maas-api-e2e-autores-c1e164-serving-cert" not found Oct 29 23:23:09.986721 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:23:09.986499 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-maas-api-tls podName:b2528a10-7f4d-4a21-8c6e-57baa9779ce7 nodeName:}" failed. No retries permitted until 2026-10-29 23:23:10.486483543 +0000 UTC m=+2437.137982052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-maas-api-tls") pod "maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" (UID: "b2528a10-7f4d-4a21-8c6e-57baa9779ce7") : secret "maas-api-e2e-autores-c1e164-serving-cert" not found Oct 29 23:23:09.986979 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:09.986865 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-openshift-service-ca\") pod \"maas-api-e2e-autores-c1e164-847ddc8596-4gnsg\" (UID: \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" Oct 29 23:23:09.988756 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:09.988729 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-metrics-certs\") pod \"maas-api-e2e-autores-c1e164-847ddc8596-4gnsg\" (UID: \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" Oct 29 23:23:10.039042 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:10.039009 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6585z\" (UniqueName: \"kubernetes.io/projected/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-kube-api-access-6585z\") pod \"maas-api-e2e-autores-c1e164-847ddc8596-4gnsg\" (UID: \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" Oct 29 23:23:10.521961 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:10.521918 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-maas-api-tls\") pod \"maas-api-e2e-autores-c1e164-847ddc8596-4gnsg\" (UID: \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" Oct 29 23:23:10.524434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:10.524410 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-maas-api-tls\") pod \"maas-api-e2e-autores-c1e164-847ddc8596-4gnsg\" (UID: \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" Oct 29 23:23:10.784298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:10.770493 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" Oct 29 23:23:11.258689 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:11.258628 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg"] Oct 29 23:23:11.264227 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:23:11.264135 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2528a10_7f4d_4a21_8c6e_57baa9779ce7.slice/crio-8560e39f973f67aadfcaff7b7790fef940bd1dbbe66a3e8cd13e2d095d9994e8 WatchSource:0}: Error finding container 8560e39f973f67aadfcaff7b7790fef940bd1dbbe66a3e8cd13e2d095d9994e8: Status 404 returned error can't find the container with id 8560e39f973f67aadfcaff7b7790fef940bd1dbbe66a3e8cd13e2d095d9994e8 Oct 29 23:23:11.265894 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:11.265869 5057 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:23:11.484736 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:11.484701 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" event={"ID":"b2528a10-7f4d-4a21-8c6e-57baa9779ce7","Type":"ContainerStarted","Data":"8560e39f973f67aadfcaff7b7790fef940bd1dbbe66a3e8cd13e2d095d9994e8"} Oct 29 23:23:12.168974 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:12.168932 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-crrbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:23:12.169142 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:12.169011 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" podUID="227efff8-cd76-476b-baa6-045e35ab4d4f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:23:12.492677 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:12.492636 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" event={"ID":"b2528a10-7f4d-4a21-8c6e-57baa9779ce7","Type":"ContainerStarted","Data":"52e0f91291f58132ad985ffa98febd226010bdfaadcdb77d6edfc24ac5331072"} Oct 29 23:23:12.493111 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:12.492705 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" Oct 29 23:23:12.542233 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:12.542154 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" podStartSLOduration=3.231038451 podStartE2EDuration="3.542139214s" totalImagesPullingTime="311.100763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:12.538502006 +0000 UTC m=+2439.190000537" watchObservedRunningTime="2026-10-29 23:23:12.542139214 +0000 UTC m=+2439.193637744" Oct 29 23:23:14.102712 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.102680 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 23:23:14.208289 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.208257 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-envoy\") pod \"227efff8-cd76-476b-baa6-045e35ab4d4f\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " Oct 29 23:23:14.208513 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.208348 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-credential-socket\") pod \"227efff8-cd76-476b-baa6-045e35ab4d4f\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " Oct 29 23:23:14.208513 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.208420 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-data\") pod \"227efff8-cd76-476b-baa6-045e35ab4d4f\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " Oct 29 23:23:14.208513 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.208500 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-podinfo\") pod \"227efff8-cd76-476b-baa6-045e35ab4d4f\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " Oct 29 23:23:14.208684 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.208666 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/227efff8-cd76-476b-baa6-045e35ab4d4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/227efff8-cd76-476b-baa6-045e35ab4d4f-istiod-ca-cert\") pod \"227efff8-cd76-476b-baa6-045e35ab4d4f\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " Oct 29 23:23:14.208747 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.208691 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-credential-socket" pod "227efff8-cd76-476b-baa6-045e35ab4d4f" (UID: "227efff8-cd76-476b-baa6-045e35ab4d4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:14.208808 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.208758 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-workload-certs\") pod \"227efff8-cd76-476b-baa6-045e35ab4d4f\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " Oct 29 23:23:14.208862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.208819 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/227efff8-cd76-476b-baa6-045e35ab4d4f-kube-api-access-vbz9s\" (UniqueName: \"kubernetes.io/projected/227efff8-cd76-476b-baa6-045e35ab4d4f-kube-api-access-vbz9s\") pod \"227efff8-cd76-476b-baa6-045e35ab4d4f\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " Oct 29 23:23:14.208918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.208882 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-token\" (UniqueName: \"kubernetes.io/projected/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-token\") pod \"227efff8-cd76-476b-baa6-045e35ab4d4f\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " Oct 29 23:23:14.208988 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.208972 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-workload-socket\") pod \"227efff8-cd76-476b-baa6-045e35ab4d4f\" (UID: \"227efff8-cd76-476b-baa6-045e35ab4d4f\") " Oct 29 23:23:14.209049 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.209004 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-data" pod "227efff8-cd76-476b-baa6-045e35ab4d4f" (UID: "227efff8-cd76-476b-baa6-045e35ab4d4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:14.209049 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.209027 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-workload-certs" pod "227efff8-cd76-476b-baa6-045e35ab4d4f" (UID: "227efff8-cd76-476b-baa6-045e35ab4d4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:14.209166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.209090 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/227efff8-cd76-476b-baa6-045e35ab4d4f-istiod-ca-cert" pod "227efff8-cd76-476b-baa6-045e35ab4d4f" (UID: "227efff8-cd76-476b-baa6-045e35ab4d4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:23:14.209305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.209285 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-workload-socket" pod "227efff8-cd76-476b-baa6-045e35ab4d4f" (UID: "227efff8-cd76-476b-baa6-045e35ab4d4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:14.210150 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.210122 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/227efff8-cd76-476b-baa6-045e35ab4d4f-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:23:14.210311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.210154 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:23:14.210311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.210169 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:23:14.210311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.210196 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:23:14.210311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.210211 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:23:14.211588 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.211564 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-token" pod "227efff8-cd76-476b-baa6-045e35ab4d4f" (UID: "227efff8-cd76-476b-baa6-045e35ab4d4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:23:14.211713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.211651 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-envoy" pod "227efff8-cd76-476b-baa6-045e35ab4d4f" (UID: "227efff8-cd76-476b-baa6-045e35ab4d4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:14.211779 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.211736 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227efff8-cd76-476b-baa6-045e35ab4d4f-kube-api-access-vbz9s" pod "227efff8-cd76-476b-baa6-045e35ab4d4f" (UID: "227efff8-cd76-476b-baa6-045e35ab4d4f"). InnerVolumeSpecName "kube-api-access-vbz9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:23:14.212083 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.212064 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-podinfo" pod "227efff8-cd76-476b-baa6-045e35ab4d4f" (UID: "227efff8-cd76-476b-baa6-045e35ab4d4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:23:14.312222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.312116 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:23:14.312222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.312155 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:23:14.312222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.312170 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbz9s\" (UniqueName: \"kubernetes.io/projected/227efff8-cd76-476b-baa6-045e35ab4d4f-kube-api-access-vbz9s\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:23:14.312458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.312229 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/227efff8-cd76-476b-baa6-045e35ab4d4f-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:23:14.509888 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.509856 5057 generic.go:356] "Generic (PLEG): container finished" podID="227efff8-cd76-476b-baa6-045e35ab4d4f" containerID="62661df807d64166aab9fa375ac64c909da5e669aef0ed4608277ba2c97dde20" exitCode=0 Oct 29 23:23:14.510072 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.509902 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" Oct 29 23:23:14.510072 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.509939 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" event={"ID":"227efff8-cd76-476b-baa6-045e35ab4d4f","Type":"ContainerDied","Data":"62661df807d64166aab9fa375ac64c909da5e669aef0ed4608277ba2c97dde20"} Oct 29 23:23:14.510072 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.509991 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd" event={"ID":"227efff8-cd76-476b-baa6-045e35ab4d4f","Type":"ContainerDied","Data":"26e917d3f668b8db76eb2b4b828730347d9c293afd3b683b932ffd4b28e8f032"} Oct 29 23:23:14.510072 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.510014 5057 scope.go:122] "RemoveContainer" containerID="62661df807d64166aab9fa375ac64c909da5e669aef0ed4608277ba2c97dde20" Oct 29 23:23:14.525140 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.525116 5057 scope.go:122] "RemoveContainer" containerID="62661df807d64166aab9fa375ac64c909da5e669aef0ed4608277ba2c97dde20" Oct 29 23:23:14.525490 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:23:14.525466 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62661df807d64166aab9fa375ac64c909da5e669aef0ed4608277ba2c97dde20\": container with ID starting with 62661df807d64166aab9fa375ac64c909da5e669aef0ed4608277ba2c97dde20 not found: ID does not exist" containerID="62661df807d64166aab9fa375ac64c909da5e669aef0ed4608277ba2c97dde20" Oct 29 23:23:14.525564 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.525502 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62661df807d64166aab9fa375ac64c909da5e669aef0ed4608277ba2c97dde20"} err="failed to get container status \"62661df807d64166aab9fa375ac64c909da5e669aef0ed4608277ba2c97dde20\": rpc error: code = NotFound desc = could not find container \"62661df807d64166aab9fa375ac64c909da5e669aef0ed4608277ba2c97dde20\": container with ID starting with 62661df807d64166aab9fa375ac64c909da5e669aef0ed4608277ba2c97dde20 not found: ID does not exist" Oct 29 23:23:14.575335 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.575251 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd"] Oct 29 23:23:14.582827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:14.582787 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-crrbd"] Oct 29 23:23:15.484265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:15.484233 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="227efff8-cd76-476b-baa6-045e35ab4d4f" path="/var/lib/kubelet/pods/227efff8-cd76-476b-baa6-045e35ab4d4f/volumes" Oct 29 23:23:18.537208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:18.536870 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" Oct 29 23:23:25.600206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:25.599382 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:23:31.491849 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.491815 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x"] Oct 29 23:23:31.493854 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.493786 5057 cpu_manager.go:414] "containerMap: removing container" podUID="227efff8-cd76-476b-baa6-045e35ab4d4f" containerName="istio-proxy" Oct 29 23:23:31.493854 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.493816 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="227efff8-cd76-476b-baa6-045e35ab4d4f" containerName="istio-proxy" Oct 29 23:23:31.494088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.493998 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="227efff8-cd76-476b-baa6-045e35ab4d4f" containerName="istio-proxy" Oct 29 23:23:31.499579 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.499552 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.559010 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.529172 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-4e2ac7-openshift-default-dockercfg-b9vvd\"" Oct 29 23:23:31.569090 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.569008 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x"] Oct 29 23:23:31.611166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.609394 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-podinfo\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.611166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.609488 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-data\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.611166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.609537 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe0b52e3-7076-420c-b899-4a8b4a3db390-istiod-ca-cert\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.611166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.609652 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-token\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.611166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.609699 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-workload-socket\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.611166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.609747 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-workload-certs\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.611166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.609962 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-credential-socket\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.611166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.610012 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5plnf\" (UniqueName: \"kubernetes.io/projected/fe0b52e3-7076-420c-b899-4a8b4a3db390-kube-api-access-5plnf\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.611166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.610093 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-envoy\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.755390 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.755303 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-workload-certs\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.755570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.755510 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-credential-socket\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.755570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.755560 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5plnf\" (UniqueName: \"kubernetes.io/projected/fe0b52e3-7076-420c-b899-4a8b4a3db390-kube-api-access-5plnf\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.755694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.755618 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-envoy\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.755694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.755669 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-podinfo\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.755808 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.755721 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-data\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.755808 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.755772 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe0b52e3-7076-420c-b899-4a8b4a3db390-istiod-ca-cert\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.755926 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.755871 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-token\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.755926 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.755918 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-workload-socket\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.756468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.756439 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-workload-socket\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.756735 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.756714 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-workload-certs\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.756993 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.756971 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-data\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.815220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.811827 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe0b52e3-7076-420c-b899-4a8b4a3db390-istiod-ca-cert\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.831211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.822533 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-credential-socket\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.913206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.907470 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-token\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.913206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.910609 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-envoy\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.913206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.911794 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-podinfo\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:31.975689 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:31.974603 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5plnf\" (UniqueName: \"kubernetes.io/projected/fe0b52e3-7076-420c-b899-4a8b4a3db390-kube-api-access-5plnf\") pod \"e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:32.136261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:32.130781 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:32.563754 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:32.563713 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x"] Oct 29 23:23:32.565063 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:23:32.565033 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0b52e3_7076_420c_b899_4a8b4a3db390.slice/crio-28d42e4aadad548bfee749437e74c8227a6404dbc9a4a637ea6a8c1f29f35064 WatchSource:0}: Error finding container 28d42e4aadad548bfee749437e74c8227a6404dbc9a4a637ea6a8c1f29f35064: Status 404 returned error can't find the container with id 28d42e4aadad548bfee749437e74c8227a6404dbc9a4a637ea6a8c1f29f35064 Oct 29 23:23:32.568584 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:32.568545 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:23:32.568703 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:32.568618 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:23:32.568703 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:32.568648 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:23:33.203219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:33.190898 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" event={"ID":"fe0b52e3-7076-420c-b899-4a8b4a3db390","Type":"ContainerStarted","Data":"d23890e7790a87be2ea6bb1cf4337883f81cf029591b064e26f77b6314ce878f"} Oct 29 23:23:33.203219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:33.190973 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" event={"ID":"fe0b52e3-7076-420c-b899-4a8b4a3db390","Type":"ContainerStarted","Data":"28d42e4aadad548bfee749437e74c8227a6404dbc9a4a637ea6a8c1f29f35064"} Oct 29 23:23:33.268471 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:33.268402 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" podStartSLOduration=2.268380076 podStartE2EDuration="2.268380076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:33.26281884 +0000 UTC m=+2459.914317361" watchObservedRunningTime="2026-10-29 23:23:33.268380076 +0000 UTC m=+2459.919878609" Oct 29 23:23:34.002440 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:23:34.002401 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:23:34.160213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:34.143298 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:34.172617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:34.172360 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:34.201810 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:34.201725 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:34.202728 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:34.202706 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:23:36.487364 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:36.487311 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:23:36.487867 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:36.487390 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" podUID="44008edf-7b81-4504-adb7-132388744c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:23:39.336215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:39.331308 5057 generic.go:356] "Generic (PLEG): container finished" podID="44008edf-7b81-4504-adb7-132388744c6e" containerID="286faffdb87fcb34fbf9536fc0e64fb232b29f1119fc5baed68bdd620efdf3f6" exitCode=0 Oct 29 23:23:39.336215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:39.331446 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" event={"ID":"44008edf-7b81-4504-adb7-132388744c6e","Type":"ContainerDied","Data":"286faffdb87fcb34fbf9536fc0e64fb232b29f1119fc5baed68bdd620efdf3f6"} Oct 29 23:23:39.336215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:39.332102 5057 scope.go:122] "RemoveContainer" containerID="286faffdb87fcb34fbf9536fc0e64fb232b29f1119fc5baed68bdd620efdf3f6" Oct 29 23:23:39.336215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:39.335632 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:23:39.336215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:39.335731 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:23:39.336215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:39.335785 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:23:40.174159 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:40.174068 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 29 23:23:40.174159 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:40.174121 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" podUID="271ab84e-ada3-4de0-8cbb-b25424172421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 29 23:23:40.340731 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:40.340697 5057 generic.go:356] "Generic (PLEG): container finished" podID="271ab84e-ada3-4de0-8cbb-b25424172421" containerID="77b1cb97d7cc92f7183a56aca462767ede0342c308d7ae2382514b54e6e52bf1" exitCode=0 Oct 29 23:23:40.341238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:40.340786 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" event={"ID":"271ab84e-ada3-4de0-8cbb-b25424172421","Type":"ContainerDied","Data":"77b1cb97d7cc92f7183a56aca462767ede0342c308d7ae2382514b54e6e52bf1"} Oct 29 23:23:40.341457 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:40.341432 5057 scope.go:122] "RemoveContainer" containerID="77b1cb97d7cc92f7183a56aca462767ede0342c308d7ae2382514b54e6e52bf1" Oct 29 23:23:40.363213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:40.344133 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" event={"ID":"44008edf-7b81-4504-adb7-132388744c6e","Type":"ContainerStarted","Data":"6b8ff8c1bd5c078b830afeae29c52ee31f127004f761d09fb42dcd9965b89e56"} Oct 29 23:23:40.363213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:40.347365 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:23:40.363213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:40.347463 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:23:40.363213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:40.347515 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:23:40.950210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:40.946905 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:23:40.950210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:40.947039 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 29 23:23:40.950210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:40.947085 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" podUID="44008edf-7b81-4504-adb7-132388744c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 29 23:23:41.365928 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:41.365825 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" event={"ID":"271ab84e-ada3-4de0-8cbb-b25424172421","Type":"ContainerStarted","Data":"8a5ff8d861ee371380177a01d33089189e85befb1d2dbd8b7e0a21a8f9fe26e4"} Oct 29 23:23:41.946210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:41.943042 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 29 23:23:41.946210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:41.943124 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" podUID="44008edf-7b81-4504-adb7-132388744c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 29 23:23:42.305396 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:42.305314 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:23:42.305553 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:42.305509 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 29 23:23:42.305597 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:42.305566 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" podUID="271ab84e-ada3-4de0-8cbb-b25424172421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 29 23:23:42.944209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:42.943622 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 29 23:23:42.944209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:42.943688 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" podUID="44008edf-7b81-4504-adb7-132388744c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 29 23:23:43.305790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:43.305698 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 29 23:23:43.306047 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:43.306018 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" podUID="271ab84e-ada3-4de0-8cbb-b25424172421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 29 23:23:43.944213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:43.942648 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 29 23:23:43.944213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:43.942717 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" podUID="44008edf-7b81-4504-adb7-132388744c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 29 23:23:44.305990 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:44.305904 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 29 23:23:44.305990 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:44.305958 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" podUID="271ab84e-ada3-4de0-8cbb-b25424172421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 29 23:23:44.943598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:44.943560 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 29 23:23:44.943948 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:44.943907 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" podUID="44008edf-7b81-4504-adb7-132388744c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 29 23:23:45.306314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:45.306226 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 29 23:23:45.306799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:45.306758 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" podUID="271ab84e-ada3-4de0-8cbb-b25424172421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 29 23:23:45.943211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:45.942609 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 29 23:23:45.943211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:45.942694 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" podUID="44008edf-7b81-4504-adb7-132388744c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 29 23:23:46.305876 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:46.305763 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 29 23:23:46.305876 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:46.305821 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" podUID="271ab84e-ada3-4de0-8cbb-b25424172421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 29 23:23:46.987212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:46.985578 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:23:46.987740 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:46.987321 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:23:46.989223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:46.988930 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:23:47.306303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:47.306222 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 29 23:23:47.306303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:47.306281 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" podUID="271ab84e-ada3-4de0-8cbb-b25424172421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 29 23:23:48.311961 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:48.311930 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:23:48.312504 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:48.312473 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:23:48.313177 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:23:48.313153 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:24:04.188244 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:24:04.186016 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:24:10.510983 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:10.510946 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr"] Oct 29 23:24:10.516378 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:10.516352 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:10.542526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:10.542498 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-fb367d-openshift-default-dockercfg-6g7pj\"" Oct 29 23:24:10.554506 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:10.554475 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr"] Oct 29 23:24:10.742216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:10.673241 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-podinfo\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:10.746253 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:10.742998 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-credential-socket\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:10.784474 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:10.784391 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-token\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:10.798212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:10.784707 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-workload-certs\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:10.799031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:10.799000 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istiod-ca-cert\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:10.799288 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:10.799272 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-workload-socket\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:10.803230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:10.799489 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-envoy\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:10.803533 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:10.803509 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdw9j\" (UniqueName: \"kubernetes.io/projected/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-kube-api-access-jdw9j\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:10.822750 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:10.822716 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-data\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:11.082748 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.082645 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-workload-socket\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:11.082748 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.082700 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-envoy\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:11.082748 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.082727 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdw9j\" (UniqueName: \"kubernetes.io/projected/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-kube-api-access-jdw9j\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:11.082976 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.082840 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-data\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:11.082976 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.082875 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-podinfo\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:11.082976 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.082972 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-credential-socket\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:11.083079 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.082989 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-token\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:11.083079 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.083029 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-workload-certs\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:11.083079 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.083053 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istiod-ca-cert\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:11.083879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.083849 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istiod-ca-cert\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:11.084041 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.083854 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-credential-socket\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:11.084139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.084116 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-workload-socket\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:11.084223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.084173 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-workload-certs\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:11.084292 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.084270 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-data\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:11.086071 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.086048 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-podinfo\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:11.086242 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.086223 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-envoy\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:11.108767 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.108738 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-token\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:11.108885 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.108811 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdw9j\" (UniqueName: \"kubernetes.io/projected/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-kube-api-access-jdw9j\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:11.177694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.177667 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:11.502071 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.502043 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr"] Oct 29 23:24:11.503915 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:24:11.503890 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895c9cf8_a426_45b5_b9b8_ea94ad9b407b.slice/crio-7630e51a0407a4de29d5b740a267ea9c9619ca88c6e4c997b2442c1a74c73800 WatchSource:0}: Error finding container 7630e51a0407a4de29d5b740a267ea9c9619ca88c6e4c997b2442c1a74c73800: Status 404 returned error can't find the container with id 7630e51a0407a4de29d5b740a267ea9c9619ca88c6e4c997b2442c1a74c73800 Oct 29 23:24:11.506855 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.506818 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:24:11.506960 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.506900 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:24:11.506960 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.506945 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:24:11.991809 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.991768 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" event={"ID":"895c9cf8-a426-45b5-b9b8-ea94ad9b407b","Type":"ContainerStarted","Data":"39e546bcbd91f4dd08cfb867bc69d11f7836bb85153515934df0e8933f3897e3"} Oct 29 23:24:11.992329 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:11.991830 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" event={"ID":"895c9cf8-a426-45b5-b9b8-ea94ad9b407b","Type":"ContainerStarted","Data":"7630e51a0407a4de29d5b740a267ea9c9619ca88c6e4c997b2442c1a74c73800"} Oct 29 23:24:12.046961 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:12.046908 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" podStartSLOduration=2.046895461 podStartE2EDuration="2.046895461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:12.04299184 +0000 UTC m=+2498.694490372" watchObservedRunningTime="2026-10-29 23:24:12.046895461 +0000 UTC m=+2498.698393991" Oct 29 23:24:12.179240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:12.178616 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:12.237208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:12.233327 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:12.999731 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:12.999701 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:13.000690 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:13.000661 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:24:21.810216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:21.810156 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:24:29.663765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:29.663726 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7"] Oct 29 23:24:29.719673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:29.717834 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7"] Oct 29 23:24:29.719673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:29.717935 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:29.828220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:29.824400 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d5e482-openshift-default-dockercfg-9mxhd\"" Oct 29 23:24:29.906410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:29.905470 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-data\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:29.906410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:29.905535 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-credential-socket\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:29.906410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:29.905625 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f466h\" (UniqueName: \"kubernetes.io/projected/00ddb9f8-8c89-451c-ae16-ae27b0da2485-kube-api-access-f466h\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:29.906410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:29.905655 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istiod-ca-cert\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:29.906410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:29.905686 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-workload-socket\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:29.906410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:29.905710 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-podinfo\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:29.906410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:29.905894 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-envoy\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:29.906410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:29.906021 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-token\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:29.906410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:29.906076 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-workload-certs\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:30.212453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:30.111224 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-data\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:30.212453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:30.111281 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-credential-socket\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:30.212453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:30.111380 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f466h\" (UniqueName: \"kubernetes.io/projected/00ddb9f8-8c89-451c-ae16-ae27b0da2485-kube-api-access-f466h\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:30.212453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:30.111407 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istiod-ca-cert\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:30.212453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:30.111440 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-workload-socket\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:30.212453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:30.111464 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-podinfo\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:30.212453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:30.111672 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-envoy\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:30.212453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:30.111953 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-token\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:30.212453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:30.112019 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-workload-certs\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:30.212453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:30.112725 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-workload-certs\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:30.212453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:30.112988 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-data\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:30.212453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:30.113242 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-credential-socket\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:30.308122 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:30.280631 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istiod-ca-cert\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:30.378642 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:30.378581 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-workload-socket\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:30.495871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:30.495683 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-token\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:30.498390 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:30.498357 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-envoy\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:30.671211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:30.670957 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f466h\" (UniqueName: \"kubernetes.io/projected/00ddb9f8-8c89-451c-ae16-ae27b0da2485-kube-api-access-f466h\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:30.693105 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:30.693042 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-podinfo\") pod \"e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:30.693653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:30.693436 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:31.284742 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:31.284354 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7"] Oct 29 23:24:31.327354 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:31.327302 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:24:31.338177 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:31.338127 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:24:31.338510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:31.338483 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:24:31.467380 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:31.467292 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" event={"ID":"00ddb9f8-8c89-451c-ae16-ae27b0da2485","Type":"ContainerStarted","Data":"33bbc51a6d67fb24492550aba216c53a7bc2c1ceacce684afa2760d7e0745279"} Oct 29 23:24:32.331369 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:32.331329 5057 patch_prober.go:29] interesting pod/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:24:32.331746 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:32.331402 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" podUID="1fcd56c2-48d5-47cc-a17f-7709d410e2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:24:32.482335 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:32.482299 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" event={"ID":"00ddb9f8-8c89-451c-ae16-ae27b0da2485","Type":"ContainerStarted","Data":"fe182763aa2258e860aec167ef10a3afd4681e8e494dec7914b71d725d651df2"} Oct 29 23:24:32.547328 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:32.547255 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" podStartSLOduration=3.547233679 podStartE2EDuration="3.547233679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:32.545811728 +0000 UTC m=+2519.197310257" watchObservedRunningTime="2026-10-29 23:24:32.547233679 +0000 UTC m=+2519.198732211" Oct 29 23:24:32.717211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:32.704575 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:32.758700 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:32.744659 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:33.494779 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:33.494749 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:33.497240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:33.496109 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:24:33.919477 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:24:33.919398 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:24:36.560858 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:36.560826 5057 generic.go:356] "Generic (PLEG): container finished" podID="1fcd56c2-48d5-47cc-a17f-7709d410e2b1" containerID="badcc38556eeb0eb1a8896d8adb3bf7cb58001749954cc94dd3fe73d36831845" exitCode=0 Oct 29 23:24:36.561490 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:36.560904 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" event={"ID":"1fcd56c2-48d5-47cc-a17f-7709d410e2b1","Type":"ContainerDied","Data":"badcc38556eeb0eb1a8896d8adb3bf7cb58001749954cc94dd3fe73d36831845"} Oct 29 23:24:36.561579 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:36.561485 5057 scope.go:122] "RemoveContainer" containerID="badcc38556eeb0eb1a8896d8adb3bf7cb58001749954cc94dd3fe73d36831845" Oct 29 23:24:36.563341 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:36.563319 5057 generic.go:356] "Generic (PLEG): container finished" podID="b7893cca-8dc6-4395-bc25-452bcd512dcd" containerID="98c68d250dbc7aaad45f78afa67d857bb15b249353ccca9a8b33c33ac1f9aa66" exitCode=0 Oct 29 23:24:36.563473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:36.563397 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" event={"ID":"b7893cca-8dc6-4395-bc25-452bcd512dcd","Type":"ContainerDied","Data":"98c68d250dbc7aaad45f78afa67d857bb15b249353ccca9a8b33c33ac1f9aa66"} Oct 29 23:24:36.563899 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:36.563881 5057 scope.go:122] "RemoveContainer" containerID="98c68d250dbc7aaad45f78afa67d857bb15b249353ccca9a8b33c33ac1f9aa66" Oct 29 23:24:36.564132 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:36.564101 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:24:36.564252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:36.564231 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:24:36.564317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:36.564288 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:24:36.565948 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:36.565920 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:24:36.566063 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:36.565986 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:24:36.566063 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:36.566020 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:24:36.617209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:36.611342 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:24:38.113613 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:38.113574 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" event={"ID":"1fcd56c2-48d5-47cc-a17f-7709d410e2b1","Type":"ContainerStarted","Data":"ee2eff3bf2dfb04550ad282099e5e96ac9c94b479a0a740c1e2b53fa35229b6a"} Oct 29 23:24:38.123299 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:38.123263 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" event={"ID":"b7893cca-8dc6-4395-bc25-452bcd512dcd","Type":"ContainerStarted","Data":"be55e009b4bffd638e8c709354e3ba8b21debdc2996d44c63e9e6a76d00920b6"} Oct 29 23:24:38.558209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:38.557542 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:24:38.576212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:38.565016 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 29 23:24:38.576212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:38.565079 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" podUID="b7893cca-8dc6-4395-bc25-452bcd512dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 29 23:24:38.599980 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:38.599946 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:24:38.600627 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:38.600606 5057 patch_prober.go:29] interesting pod/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 29 23:24:38.600699 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:38.600653 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" podUID="1fcd56c2-48d5-47cc-a17f-7709d410e2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 29 23:24:39.559209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:39.558582 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 29 23:24:39.559209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:39.558666 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" podUID="b7893cca-8dc6-4395-bc25-452bcd512dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 29 23:24:39.601100 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:39.601064 5057 patch_prober.go:29] interesting pod/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 29 23:24:39.601328 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:39.601118 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" podUID="1fcd56c2-48d5-47cc-a17f-7709d410e2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 29 23:24:40.583426 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:40.573600 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 29 23:24:40.583426 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:40.573668 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" podUID="b7893cca-8dc6-4395-bc25-452bcd512dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 29 23:24:40.600201 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:40.600142 5057 patch_prober.go:29] interesting pod/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 29 23:24:40.600403 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:40.600226 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" podUID="1fcd56c2-48d5-47cc-a17f-7709d410e2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 29 23:24:41.593207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:41.572174 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 29 23:24:41.593207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:41.572252 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" podUID="b7893cca-8dc6-4395-bc25-452bcd512dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 29 23:24:41.609301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:41.609260 5057 patch_prober.go:29] interesting pod/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 29 23:24:41.609506 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:41.609328 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" podUID="1fcd56c2-48d5-47cc-a17f-7709d410e2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 29 23:24:42.557211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:42.556631 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 29 23:24:42.557211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:42.556700 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" podUID="b7893cca-8dc6-4395-bc25-452bcd512dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 29 23:24:42.600105 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:42.600074 5057 patch_prober.go:29] interesting pod/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 29 23:24:42.600620 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:42.600281 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" podUID="1fcd56c2-48d5-47cc-a17f-7709d410e2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 29 23:24:43.596796 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:43.596621 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 29 23:24:43.596796 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:43.596697 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" podUID="b7893cca-8dc6-4395-bc25-452bcd512dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 29 23:24:43.615238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:43.604457 5057 patch_prober.go:29] interesting pod/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 29 23:24:43.615238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:43.604522 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" podUID="1fcd56c2-48d5-47cc-a17f-7709d410e2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 29 23:24:44.562774 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:44.562739 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:24:44.563255 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:44.563236 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:24:44.564007 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:44.563989 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:24:44.606609 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:44.606570 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:24:44.606876 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:44.606859 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:24:44.607703 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:24:44.607674 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:25:03.917013 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:25:03.916978 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:25:08.937883 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:08.937846 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg"] Oct 29 23:25:08.938849 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:08.938811 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" podUID="b2528a10-7f4d-4a21-8c6e-57baa9779ce7" containerName="maas-api" containerID="cri-o://52e0f91291f58132ad985ffa98febd226010bdfaadcdb77d6edfc24ac5331072" gracePeriod=30 Oct 29 23:25:09.813397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:09.812973 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" Oct 29 23:25:09.819903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:09.819871 5057 generic.go:356] "Generic (PLEG): container finished" podID="b2528a10-7f4d-4a21-8c6e-57baa9779ce7" containerID="52e0f91291f58132ad985ffa98febd226010bdfaadcdb77d6edfc24ac5331072" exitCode=0 Oct 29 23:25:09.820141 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:09.819916 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" Oct 29 23:25:09.820240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:09.819932 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" event={"ID":"b2528a10-7f4d-4a21-8c6e-57baa9779ce7","Type":"ContainerDied","Data":"52e0f91291f58132ad985ffa98febd226010bdfaadcdb77d6edfc24ac5331072"} Oct 29 23:25:09.820316 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:09.820284 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg" event={"ID":"b2528a10-7f4d-4a21-8c6e-57baa9779ce7","Type":"ContainerDied","Data":"8560e39f973f67aadfcaff7b7790fef940bd1dbbe66a3e8cd13e2d095d9994e8"} Oct 29 23:25:09.820373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:09.820317 5057 scope.go:122] "RemoveContainer" containerID="52e0f91291f58132ad985ffa98febd226010bdfaadcdb77d6edfc24ac5331072" Oct 29 23:25:09.835346 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:09.835321 5057 scope.go:122] "RemoveContainer" containerID="52e0f91291f58132ad985ffa98febd226010bdfaadcdb77d6edfc24ac5331072" Oct 29 23:25:09.835888 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:25:09.835857 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52e0f91291f58132ad985ffa98febd226010bdfaadcdb77d6edfc24ac5331072\": container with ID starting with 52e0f91291f58132ad985ffa98febd226010bdfaadcdb77d6edfc24ac5331072 not found: ID does not exist" containerID="52e0f91291f58132ad985ffa98febd226010bdfaadcdb77d6edfc24ac5331072" Oct 29 23:25:09.835999 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:09.835900 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52e0f91291f58132ad985ffa98febd226010bdfaadcdb77d6edfc24ac5331072"} err="failed to get container status \"52e0f91291f58132ad985ffa98febd226010bdfaadcdb77d6edfc24ac5331072\": rpc error: code = NotFound desc = could not find container \"52e0f91291f58132ad985ffa98febd226010bdfaadcdb77d6edfc24ac5331072\": container with ID starting with 52e0f91291f58132ad985ffa98febd226010bdfaadcdb77d6edfc24ac5331072 not found: ID does not exist" Oct 29 23:25:09.907180 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:09.907146 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-kube-api-access-6585z\" (UniqueName: \"kubernetes.io/projected/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-kube-api-access-6585z\") pod \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\" (UID: \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\") " Oct 29 23:25:09.907391 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:09.907215 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-metrics-certs\") pod \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\" (UID: \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\") " Oct 29 23:25:09.907391 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:09.907295 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-maas-api-tls\") pod \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\" (UID: \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\") " Oct 29 23:25:09.907391 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:09.907336 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-openshift-service-ca\") pod \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\" (UID: \"b2528a10-7f4d-4a21-8c6e-57baa9779ce7\") " Oct 29 23:25:09.907799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:09.907769 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-openshift-service-ca" pod "b2528a10-7f4d-4a21-8c6e-57baa9779ce7" (UID: "b2528a10-7f4d-4a21-8c6e-57baa9779ce7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:09.908403 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:09.908376 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:25:09.909444 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:09.909421 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-maas-api-tls" pod "b2528a10-7f4d-4a21-8c6e-57baa9779ce7" (UID: "b2528a10-7f4d-4a21-8c6e-57baa9779ce7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:09.909531 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:09.909440 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-metrics-certs" pod "b2528a10-7f4d-4a21-8c6e-57baa9779ce7" (UID: "b2528a10-7f4d-4a21-8c6e-57baa9779ce7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:09.909531 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:09.909458 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-kube-api-access-6585z" pod "b2528a10-7f4d-4a21-8c6e-57baa9779ce7" (UID: "b2528a10-7f4d-4a21-8c6e-57baa9779ce7"). InnerVolumeSpecName "kube-api-access-6585z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:10.009991 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:10.009958 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6585z\" (UniqueName: \"kubernetes.io/projected/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-kube-api-access-6585z\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:25:10.009991 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:10.009987 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:25:10.009991 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:10.009999 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2528a10-7f4d-4a21-8c6e-57baa9779ce7-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:25:10.196291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:10.196246 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg"] Oct 29 23:25:10.233793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:10.233748 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c1e164-847ddc8596-4gnsg"] Oct 29 23:25:11.483559 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:11.483522 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2528a10-7f4d-4a21-8c6e-57baa9779ce7" path="/var/lib/kubelet/pods/b2528a10-7f4d-4a21-8c6e-57baa9779ce7/volumes" Oct 29 23:25:17.667346 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:17.667266 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt"] Oct 29 23:25:17.668678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:17.667524 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" podUID="1fcd56c2-48d5-47cc-a17f-7709d410e2b1" containerName="istio-proxy" containerID="cri-o://ee2eff3bf2dfb04550ad282099e5e96ac9c94b479a0a740c1e2b53fa35229b6a" gracePeriod=30 Oct 29 23:25:20.085917 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.085882 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4"] Oct 29 23:25:20.087760 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.087733 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b2528a10-7f4d-4a21-8c6e-57baa9779ce7" containerName="maas-api" Oct 29 23:25:20.087909 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.087896 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2528a10-7f4d-4a21-8c6e-57baa9779ce7" containerName="maas-api" Oct 29 23:25:20.088275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.088261 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2528a10-7f4d-4a21-8c6e-57baa9779ce7" containerName="maas-api" Oct 29 23:25:20.093101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.093080 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.120617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.120562 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4"] Oct 29 23:25:20.300456 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.153392 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-45036b-openshift-default-dockercfg-pblxp\"" Oct 29 23:25:20.430260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.427651 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6508b22-be11-4d60-80bd-42a88a299853-istio-token\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.430260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.427769 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-credential-socket\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.430260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.427821 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-istio-envoy\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.430260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.427867 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-workload-certs\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.430260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.427967 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6508b22-be11-4d60-80bd-42a88a299853-istiod-ca-cert\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.430260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.428115 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6508b22-be11-4d60-80bd-42a88a299853-istio-podinfo\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.430260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.428155 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-workload-socket\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.430260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.428223 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-istio-data\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.430260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.428298 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f946r\" (UniqueName: \"kubernetes.io/projected/c6508b22-be11-4d60-80bd-42a88a299853-kube-api-access-f946r\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.544220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.539323 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f946r\" (UniqueName: \"kubernetes.io/projected/c6508b22-be11-4d60-80bd-42a88a299853-kube-api-access-f946r\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.559532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.554681 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6508b22-be11-4d60-80bd-42a88a299853-istio-token\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.559532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.554862 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-credential-socket\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.559532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.554920 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-istio-envoy\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.559532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.555008 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-workload-certs\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.559532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.555174 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6508b22-be11-4d60-80bd-42a88a299853-istiod-ca-cert\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.559532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.555494 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6508b22-be11-4d60-80bd-42a88a299853-istio-podinfo\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.559532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.555539 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-workload-socket\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.559532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.555653 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-istio-data\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.592733 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.561345 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-istio-data\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.618617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.618574 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-credential-socket\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.669247 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.628974 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-workload-certs\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.715138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.640285 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6508b22-be11-4d60-80bd-42a88a299853-istiod-ca-cert\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.717299 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.675577 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-workload-socket\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.717465 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.702020 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f946r\" (UniqueName: \"kubernetes.io/projected/c6508b22-be11-4d60-80bd-42a88a299853-kube-api-access-f946r\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.717605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.716105 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6508b22-be11-4d60-80bd-42a88a299853-istio-token\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.757786 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.756573 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-istio-envoy\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.779854 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.779819 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6508b22-be11-4d60-80bd-42a88a299853-istio-podinfo\") pod \"e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:20.977716 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:20.977626 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:21.457672 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:25:21.457634 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6508b22_be11_4d60_80bd_42a88a299853.slice/crio-a2fe8d8718ea6dcbc49659a1017125876c3581e8f3f1c8dfa5909b3c6cef2769 WatchSource:0}: Error finding container a2fe8d8718ea6dcbc49659a1017125876c3581e8f3f1c8dfa5909b3c6cef2769: Status 404 returned error can't find the container with id a2fe8d8718ea6dcbc49659a1017125876c3581e8f3f1c8dfa5909b3c6cef2769 Oct 29 23:25:21.460050 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:21.460011 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4"] Oct 29 23:25:21.462321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:21.462287 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:25:21.462428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:21.462363 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:25:21.462428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:21.462396 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:25:22.005963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:22.005932 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" event={"ID":"c6508b22-be11-4d60-80bd-42a88a299853","Type":"ContainerStarted","Data":"2d557413580e7e4deb2404281cec8352a50820c901754479b16340047f6134fc"} Oct 29 23:25:22.006145 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:22.005981 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" event={"ID":"c6508b22-be11-4d60-80bd-42a88a299853","Type":"ContainerStarted","Data":"a2fe8d8718ea6dcbc49659a1017125876c3581e8f3f1c8dfa5909b3c6cef2769"} Oct 29 23:25:22.116795 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:22.116741 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" podStartSLOduration=2.116721695 podStartE2EDuration="2.116721695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:22.098557093 +0000 UTC m=+2568.750055625" watchObservedRunningTime="2026-10-29 23:25:22.116721695 +0000 UTC m=+2568.768220228" Oct 29 23:25:22.981332 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:22.981252 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:22.989602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:22.989574 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:23.020424 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.020393 5057 generic.go:356] "Generic (PLEG): container finished" podID="1fcd56c2-48d5-47cc-a17f-7709d410e2b1" containerID="ee2eff3bf2dfb04550ad282099e5e96ac9c94b479a0a740c1e2b53fa35229b6a" exitCode=0 Oct 29 23:25:23.020586 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.020463 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" event={"ID":"1fcd56c2-48d5-47cc-a17f-7709d410e2b1","Type":"ContainerDied","Data":"ee2eff3bf2dfb04550ad282099e5e96ac9c94b479a0a740c1e2b53fa35229b6a"} Oct 29 23:25:23.020586 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.020518 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" event={"ID":"1fcd56c2-48d5-47cc-a17f-7709d410e2b1","Type":"ContainerDied","Data":"d9193febdb7bfac76f9dcfc497d7f072beeb9767c6765a6363ebb05a454701d5"} Oct 29 23:25:23.020586 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.020539 5057 scope.go:122] "RemoveContainer" containerID="badcc38556eeb0eb1a8896d8adb3bf7cb58001749954cc94dd3fe73d36831845" Oct 29 23:25:23.020732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.020597 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9193febdb7bfac76f9dcfc497d7f072beeb9767c6765a6363ebb05a454701d5" Oct 29 23:25:23.021075 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.021059 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:23.056162 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.055900 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:25:23.084464 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.084436 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:25:23.211290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.201318 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-workload-socket\") pod \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " Oct 29 23:25:23.211290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.201378 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-token\" (UniqueName: \"kubernetes.io/projected/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-token\") pod \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " Oct 29 23:25:23.211290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.201432 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-podinfo\") pod \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " Oct 29 23:25:23.211290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.201475 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-kube-api-access-fhbk6\" (UniqueName: \"kubernetes.io/projected/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-kube-api-access-fhbk6\") pod \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " Oct 29 23:25:23.211290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.201520 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-credential-socket\") pod \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " Oct 29 23:25:23.211290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.201578 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-data\") pod \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " Oct 29 23:25:23.211290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.201658 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-envoy\") pod \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " Oct 29 23:25:23.211290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.201709 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istiod-ca-cert\") pod \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " Oct 29 23:25:23.211290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.201735 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-workload-certs\") pod \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\" (UID: \"1fcd56c2-48d5-47cc-a17f-7709d410e2b1\") " Oct 29 23:25:23.211290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.205753 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istiod-ca-cert" pod "1fcd56c2-48d5-47cc-a17f-7709d410e2b1" (UID: "1fcd56c2-48d5-47cc-a17f-7709d410e2b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:23.211290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.206016 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-workload-certs" pod "1fcd56c2-48d5-47cc-a17f-7709d410e2b1" (UID: "1fcd56c2-48d5-47cc-a17f-7709d410e2b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:23.211290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.206324 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-credential-socket" pod "1fcd56c2-48d5-47cc-a17f-7709d410e2b1" (UID: "1fcd56c2-48d5-47cc-a17f-7709d410e2b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:23.211290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.206496 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-data" pod "1fcd56c2-48d5-47cc-a17f-7709d410e2b1" (UID: "1fcd56c2-48d5-47cc-a17f-7709d410e2b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:23.211290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.209971 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-podinfo" pod "1fcd56c2-48d5-47cc-a17f-7709d410e2b1" (UID: "1fcd56c2-48d5-47cc-a17f-7709d410e2b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:25:23.211290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.210243 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-token" pod "1fcd56c2-48d5-47cc-a17f-7709d410e2b1" (UID: "1fcd56c2-48d5-47cc-a17f-7709d410e2b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:23.212056 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.212017 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-workload-socket" pod "1fcd56c2-48d5-47cc-a17f-7709d410e2b1" (UID: "1fcd56c2-48d5-47cc-a17f-7709d410e2b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:23.299210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.285339 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-envoy" pod "1fcd56c2-48d5-47cc-a17f-7709d410e2b1" (UID: "1fcd56c2-48d5-47cc-a17f-7709d410e2b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:23.300493 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.300432 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-kube-api-access-fhbk6" pod "1fcd56c2-48d5-47cc-a17f-7709d410e2b1" (UID: "1fcd56c2-48d5-47cc-a17f-7709d410e2b1"). InnerVolumeSpecName "kube-api-access-fhbk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:23.345219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.334038 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:25:23.345219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.334074 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:25:23.345219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.334089 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:25:23.345219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.334104 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhbk6\" (UniqueName: \"kubernetes.io/projected/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-kube-api-access-fhbk6\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:25:23.345219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.334119 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:25:23.345219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.334131 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:25:23.345219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.334143 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:25:23.345219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.334155 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:25:23.345219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:23.334167 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fcd56c2-48d5-47cc-a17f-7709d410e2b1-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:25:24.060058 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:24.060030 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt" Oct 29 23:25:24.176442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:24.176410 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt"] Oct 29 23:25:24.180690 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:24.180665 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-c1e164-openshift-default-86796d9dcf-9zxlt"] Oct 29 23:25:25.484314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:25.484281 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fcd56c2-48d5-47cc-a17f-7709d410e2b1" path="/var/lib/kubelet/pods/1fcd56c2-48d5-47cc-a17f-7709d410e2b1/volumes" Oct 29 23:25:30.139774 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:30.139669 5057 generic.go:356] "Generic (PLEG): container finished" podID="fe0b52e3-7076-420c-b899-4a8b4a3db390" containerID="d23890e7790a87be2ea6bb1cf4337883f81cf029591b064e26f77b6314ce878f" exitCode=0 Oct 29 23:25:30.139774 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:30.139753 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" event={"ID":"fe0b52e3-7076-420c-b899-4a8b4a3db390","Type":"ContainerDied","Data":"d23890e7790a87be2ea6bb1cf4337883f81cf029591b064e26f77b6314ce878f"} Oct 29 23:25:30.140494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:30.140456 5057 scope.go:122] "RemoveContainer" containerID="d23890e7790a87be2ea6bb1cf4337883f81cf029591b064e26f77b6314ce878f" Oct 29 23:25:30.144357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:30.144088 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:25:30.144357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:30.144198 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:25:30.144357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:30.144249 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:25:31.154258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:31.154209 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" event={"ID":"fe0b52e3-7076-420c-b899-4a8b4a3db390","Type":"ContainerStarted","Data":"3eedf5d6f7cb7cca0d2451364457707723a72f680ea0c729d6d43fcd3fdf422f"} Oct 29 23:25:32.138211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:32.133469 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:25:32.152365 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:32.152327 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 29 23:25:32.152551 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:32.152389 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" podUID="fe0b52e3-7076-420c-b899-4a8b4a3db390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 29 23:25:33.136334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:33.136299 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 29 23:25:33.136996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:33.136950 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" podUID="fe0b52e3-7076-420c-b899-4a8b4a3db390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 29 23:25:33.931442 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:25:33.931372 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:25:34.133538 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:34.133499 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 29 23:25:34.133805 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:34.133777 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" podUID="fe0b52e3-7076-420c-b899-4a8b4a3db390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 29 23:25:34.202083 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:34.201992 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:25:35.146945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:35.146908 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 29 23:25:35.147730 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:35.147698 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" podUID="fe0b52e3-7076-420c-b899-4a8b4a3db390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 29 23:25:36.138711 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:36.138675 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 29 23:25:36.139332 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:36.139296 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" podUID="fe0b52e3-7076-420c-b899-4a8b4a3db390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 29 23:25:37.267955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:37.267923 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:25:37.271846 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:37.271810 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:25:49.699551 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.699515 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6"] Oct 29 23:25:49.700440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.700419 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1fcd56c2-48d5-47cc-a17f-7709d410e2b1" containerName="istio-proxy" Oct 29 23:25:49.700440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.700436 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fcd56c2-48d5-47cc-a17f-7709d410e2b1" containerName="istio-proxy" Oct 29 23:25:49.700650 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.700466 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1fcd56c2-48d5-47cc-a17f-7709d410e2b1" containerName="istio-proxy" Oct 29 23:25:49.700650 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.700471 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fcd56c2-48d5-47cc-a17f-7709d410e2b1" containerName="istio-proxy" Oct 29 23:25:49.700650 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.700603 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fcd56c2-48d5-47cc-a17f-7709d410e2b1" containerName="istio-proxy" Oct 29 23:25:49.700650 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.700620 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fcd56c2-48d5-47cc-a17f-7709d410e2b1" containerName="istio-proxy" Oct 29 23:25:49.711248 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.707027 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" Oct 29 23:25:49.711610 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.711585 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6"] Oct 29 23:25:49.738829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.738798 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 29 23:25:49.739034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.738819 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-76ad5472ea54\"" Oct 29 23:25:49.739136 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.738821 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 29 23:25:49.739136 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.738848 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 29 23:25:49.739136 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.738891 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-prms8\"" Oct 29 23:25:49.739136 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.738977 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 29 23:25:49.873566 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.872677 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-plugins-config-volume\") pod \"payload-processing-external-model-844cdcdcd7-jxft6\" (UID: \"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0\") " pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" Oct 29 23:25:49.873566 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.872807 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-provider-credentials\") pod \"payload-processing-external-model-844cdcdcd7-jxft6\" (UID: \"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0\") " pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" Oct 29 23:25:49.873566 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.872930 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-routing-overlay-volume\") pod \"payload-processing-external-model-844cdcdcd7-jxft6\" (UID: \"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0\") " pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" Oct 29 23:25:49.873566 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.872986 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n7jc\" (UniqueName: \"kubernetes.io/projected/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-kube-api-access-4n7jc\") pod \"payload-processing-external-model-844cdcdcd7-jxft6\" (UID: \"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0\") " pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" Oct 29 23:25:49.978257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.975750 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-routing-overlay-volume\") pod \"payload-processing-external-model-844cdcdcd7-jxft6\" (UID: \"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0\") " pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" Oct 29 23:25:49.978257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.975829 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n7jc\" (UniqueName: \"kubernetes.io/projected/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-kube-api-access-4n7jc\") pod \"payload-processing-external-model-844cdcdcd7-jxft6\" (UID: \"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0\") " pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" Oct 29 23:25:49.978257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.976013 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-plugins-config-volume\") pod \"payload-processing-external-model-844cdcdcd7-jxft6\" (UID: \"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0\") " pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" Oct 29 23:25:49.978257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.976107 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-provider-credentials\") pod \"payload-processing-external-model-844cdcdcd7-jxft6\" (UID: \"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0\") " pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" Oct 29 23:25:49.988261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.983250 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-routing-overlay-volume\") pod \"payload-processing-external-model-844cdcdcd7-jxft6\" (UID: \"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0\") " pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" Oct 29 23:25:49.989275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.989248 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-provider-credentials\") pod \"payload-processing-external-model-844cdcdcd7-jxft6\" (UID: \"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0\") " pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" Oct 29 23:25:49.994265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:49.989290 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-plugins-config-volume\") pod \"payload-processing-external-model-844cdcdcd7-jxft6\" (UID: \"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0\") " pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" Oct 29 23:25:50.069208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:50.064460 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n7jc\" (UniqueName: \"kubernetes.io/projected/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-kube-api-access-4n7jc\") pod \"payload-processing-external-model-844cdcdcd7-jxft6\" (UID: \"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0\") " pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" Oct 29 23:25:50.328862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:50.328774 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" Oct 29 23:25:50.701730 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:25:50.701505 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7660eca1_9fd8_4b74_9fb1_f3ce4d5c16a0.slice/crio-26e4ef912dab9e8351f072276cfb4831c314ee6624e4f6df60756a4deafbac19 WatchSource:0}: Error finding container 26e4ef912dab9e8351f072276cfb4831c314ee6624e4f6df60756a4deafbac19: Status 404 returned error can't find the container with id 26e4ef912dab9e8351f072276cfb4831c314ee6624e4f6df60756a4deafbac19 Oct 29 23:25:50.702365 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:50.702258 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6"] Oct 29 23:25:51.654669 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:51.654631 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" event={"ID":"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0","Type":"ContainerStarted","Data":"bb2358e307b54e12411cc8379929467c96da9d40b531d1afe7ba182a893a6d69"} Oct 29 23:25:51.654865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:51.654684 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" event={"ID":"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0","Type":"ContainerStarted","Data":"26e4ef912dab9e8351f072276cfb4831c314ee6624e4f6df60756a4deafbac19"} Oct 29 23:25:51.654865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:51.654852 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" Oct 29 23:25:51.704988 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:25:51.703373 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" podStartSLOduration=2.703353035 podStartE2EDuration="2.703353035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:51.701282428 +0000 UTC m=+2598.352780958" watchObservedRunningTime="2026-10-29 23:25:51.703353035 +0000 UTC m=+2598.354851569" Oct 29 23:26:02.665900 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:02.665867 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" Oct 29 23:26:03.915013 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:26:03.914976 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:26:10.781207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:10.775589 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:26:24.317875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.317837 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r"] Oct 29 23:26:24.326580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.326557 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.343829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.343804 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h76hm\"" Oct 29 23:26:24.367339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.367301 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r"] Oct 29 23:26:24.476394 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.472261 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.476394 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.472352 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.476394 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.472485 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.476394 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.472516 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.476394 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.472539 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.476394 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.472595 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z79dk\" (UniqueName: \"kubernetes.io/projected/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-kube-api-access-z79dk\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.476394 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.472672 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.476394 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.472705 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.476394 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.472736 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.576331 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.576236 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.576331 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.576312 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.576604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.576539 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.576604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.576575 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.576604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.576599 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.576790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.576653 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79dk\" (UniqueName: \"kubernetes.io/projected/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-kube-api-access-z79dk\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.576790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.576753 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.576921 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.576789 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.576921 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.576839 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.577050 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.576993 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.577433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.577228 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.577433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.577228 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.577433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.577413 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.577704 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.577466 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.579410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.579382 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.579410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.579398 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.602122 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.602084 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.602927 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.602899 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z79dk\" (UniqueName: \"kubernetes.io/projected/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-kube-api-access-z79dk\") pod \"maas-default-gateway-openshift-default-55fbd65575-x4v6r\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:24.647242 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:24.646990 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:25.005234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:25.003454 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r"] Oct 29 23:26:25.008789 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:25.008731 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:26:25.008918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:25.008888 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:26:25.008992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:25.008973 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:26:25.431403 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:25.431367 5057 generic.go:356] "Generic (PLEG): container finished" podID="00ddb9f8-8c89-451c-ae16-ae27b0da2485" containerID="fe182763aa2258e860aec167ef10a3afd4681e8e494dec7914b71d725d651df2" exitCode=0 Oct 29 23:26:25.431942 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:25.431443 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" event={"ID":"00ddb9f8-8c89-451c-ae16-ae27b0da2485","Type":"ContainerDied","Data":"fe182763aa2258e860aec167ef10a3afd4681e8e494dec7914b71d725d651df2"} Oct 29 23:26:25.432023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:25.431957 5057 scope.go:122] "RemoveContainer" containerID="fe182763aa2258e860aec167ef10a3afd4681e8e494dec7914b71d725d651df2" Oct 29 23:26:25.433875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:25.433849 5057 generic.go:356] "Generic (PLEG): container finished" podID="895c9cf8-a426-45b5-b9b8-ea94ad9b407b" containerID="39e546bcbd91f4dd08cfb867bc69d11f7836bb85153515934df0e8933f3897e3" exitCode=0 Oct 29 23:26:25.433993 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:25.433928 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" event={"ID":"895c9cf8-a426-45b5-b9b8-ea94ad9b407b","Type":"ContainerDied","Data":"39e546bcbd91f4dd08cfb867bc69d11f7836bb85153515934df0e8933f3897e3"} Oct 29 23:26:25.434378 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:25.434356 5057 scope.go:122] "RemoveContainer" containerID="39e546bcbd91f4dd08cfb867bc69d11f7836bb85153515934df0e8933f3897e3" Oct 29 23:26:25.434776 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:25.434738 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:26:25.434887 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:25.434831 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:26:25.434952 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:25.434898 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:26:25.437519 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:25.436873 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:26:25.437519 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:25.436956 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:26:25.437519 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:25.437002 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:26:25.437776 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:25.437672 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" event={"ID":"e5ee3546-b9fa-49f9-a7b1-c110932ebb10","Type":"ContainerStarted","Data":"f43af5c9cd246f7adfbc1d4ad3d08d0b92b7a172751f9a186c7e9aefec4191c7"} Oct 29 23:26:25.437776 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:25.437713 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" event={"ID":"e5ee3546-b9fa-49f9-a7b1-c110932ebb10","Type":"ContainerStarted","Data":"303e38f689a2f80acdd878d05f48487749c7cee45907d27f8af72e95dd35a92a"} Oct 29 23:26:25.661031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:25.653732 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:25.661031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:25.659107 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" podStartSLOduration=1.659083958 podStartE2EDuration="1.659083958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:25.63355665 +0000 UTC m=+2632.285055182" watchObservedRunningTime="2026-10-29 23:26:25.659083958 +0000 UTC m=+2632.310582489" Oct 29 23:26:26.472577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:26.472535 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" event={"ID":"00ddb9f8-8c89-451c-ae16-ae27b0da2485","Type":"ContainerStarted","Data":"8ba322697e62b2fb98ae9280de624a2418890b7bd6b3206d95d23f3afe7a7ee8"} Oct 29 23:26:26.476949 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:26.476914 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" event={"ID":"895c9cf8-a426-45b5-b9b8-ea94ad9b407b","Type":"ContainerStarted","Data":"91e19ca045ac4f61702f9f1624d8db945a70a09737883b3f2683be7c3f49f3a8"} Oct 29 23:26:26.662279 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:26.661898 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-x4v6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 29 23:26:26.662279 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:26.661962 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" podUID="e5ee3546-b9fa-49f9-a7b1-c110932ebb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 29 23:26:26.701212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:26.700661 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:26:26.701212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:26.700811 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 29 23:26:26.701212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:26.700865 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" podUID="00ddb9f8-8c89-451c-ae16-ae27b0da2485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 29 23:26:27.182214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:27.181731 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:26:27.182214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:27.181869 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 29 23:26:27.182214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:27.181915 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" podUID="895c9cf8-a426-45b5-b9b8-ea94ad9b407b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 29 23:26:27.650264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:27.649597 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-x4v6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 29 23:26:27.650264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:27.649663 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" podUID="e5ee3546-b9fa-49f9-a7b1-c110932ebb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 29 23:26:27.696209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:27.695655 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 29 23:26:27.696209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:27.695725 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" podUID="00ddb9f8-8c89-451c-ae16-ae27b0da2485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 29 23:26:28.038041 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:28.037655 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:26:28.183212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:28.182718 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 29 23:26:28.183212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:28.182789 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" podUID="895c9cf8-a426-45b5-b9b8-ea94ad9b407b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 29 23:26:28.649789 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:28.649746 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-x4v6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 29 23:26:28.650105 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:28.650058 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" podUID="e5ee3546-b9fa-49f9-a7b1-c110932ebb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 29 23:26:28.695373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:28.695339 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 29 23:26:28.696294 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:28.696260 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" podUID="00ddb9f8-8c89-451c-ae16-ae27b0da2485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 29 23:26:29.179212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:29.178907 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 29 23:26:29.179212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:29.178968 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" podUID="895c9cf8-a426-45b5-b9b8-ea94ad9b407b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 29 23:26:29.648266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:29.647395 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-x4v6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 29 23:26:29.648266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:29.647456 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" podUID="e5ee3546-b9fa-49f9-a7b1-c110932ebb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 29 23:26:29.703728 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:29.703689 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 29 23:26:29.720948 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:29.720907 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" podUID="00ddb9f8-8c89-451c-ae16-ae27b0da2485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 29 23:26:30.133822 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:30.132549 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:26:30.180826 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:30.180788 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 29 23:26:30.181118 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:30.181088 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" podUID="895c9cf8-a426-45b5-b9b8-ea94ad9b407b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 29 23:26:30.669272 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:30.660314 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-x4v6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 29 23:26:30.669272 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:30.660380 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" podUID="e5ee3546-b9fa-49f9-a7b1-c110932ebb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 29 23:26:30.706219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:30.705530 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 29 23:26:30.706219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:30.705594 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" podUID="00ddb9f8-8c89-451c-ae16-ae27b0da2485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 29 23:26:31.129255 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:31.128305 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:26:31.155209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:31.129856 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 29 23:26:31.282062 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:31.281952 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 29 23:26:31.282390 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:31.282317 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" podUID="895c9cf8-a426-45b5-b9b8-ea94ad9b407b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 29 23:26:31.286153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:31.286066 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:26:31.287041 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:31.286977 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:26:31.479722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:31.479661 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:26:31.777318 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:31.777230 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-x4v6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 29 23:26:31.777931 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:31.777893 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" podUID="e5ee3546-b9fa-49f9-a7b1-c110932ebb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 29 23:26:32.105267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:32.105096 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 29 23:26:32.105267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:32.105175 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" podUID="00ddb9f8-8c89-451c-ae16-ae27b0da2485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 29 23:26:32.139209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:32.138633 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:26:32.299324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:32.299288 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 29 23:26:32.299584 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:32.299555 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" podUID="895c9cf8-a426-45b5-b9b8-ea94ad9b407b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 29 23:26:32.403277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:32.380711 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:26:32.403277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:32.380784 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:26:32.485345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:32.485299 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:26:32.486787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:32.486748 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:26:32.594668 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:32.594624 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:26:32.718211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:32.717793 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-x4v6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 29 23:26:32.718211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:32.717855 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" podUID="e5ee3546-b9fa-49f9-a7b1-c110932ebb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 29 23:26:32.743212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:32.719056 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 29 23:26:32.743212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:32.719102 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" podUID="00ddb9f8-8c89-451c-ae16-ae27b0da2485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 29 23:26:32.786547 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:32.786501 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:26:33.017730 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:33.017619 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:26:33.267210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:33.233644 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 29 23:26:33.267210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:33.233712 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" podUID="895c9cf8-a426-45b5-b9b8-ea94ad9b407b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 29 23:26:33.688212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:33.667594 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-x4v6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 29 23:26:33.688212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:33.667655 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" podUID="e5ee3546-b9fa-49f9-a7b1-c110932ebb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 29 23:26:33.724613 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:33.716974 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:26:33.741217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:33.734900 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 29 23:26:33.741217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:33.734951 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" podUID="00ddb9f8-8c89-451c-ae16-ae27b0da2485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 29 23:26:34.116627 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:26:34.116528 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:26:34.178421 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:34.178383 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 29 23:26:34.178772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:34.178740 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" podUID="895c9cf8-a426-45b5-b9b8-ea94ad9b407b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 29 23:26:34.672676 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:34.670757 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-x4v6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 29 23:26:34.672676 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:34.670824 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" podUID="e5ee3546-b9fa-49f9-a7b1-c110932ebb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 29 23:26:34.698664 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:34.698579 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 29 23:26:34.698894 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:34.698657 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" podUID="00ddb9f8-8c89-451c-ae16-ae27b0da2485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 29 23:26:35.373924 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:35.372742 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:26:35.375716 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:35.375647 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:26:35.663265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:35.658769 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-x4v6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 29 23:26:35.663265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:35.658839 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" podUID="e5ee3546-b9fa-49f9-a7b1-c110932ebb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 29 23:26:35.851303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:35.848865 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:26:35.856471 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:35.856445 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:26:37.489223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:37.489179 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:37.489681 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:37.489583 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:37.490697 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:37.490678 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:26:39.368745 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.368709 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k"] Oct 29 23:26:39.373347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.373318 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.409698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.409664 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k"] Oct 29 23:26:39.544784 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.544750 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.544784 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.544793 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.545023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.544827 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.545023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.544891 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.545023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.544947 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.545023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.544985 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.545167 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.545082 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.545228 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.545209 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wm8r\" (UniqueName: \"kubernetes.io/projected/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-kube-api-access-9wm8r\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.545273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.545248 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.646479 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.646388 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.646479 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.646431 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.646479 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.646456 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.646791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.646585 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.646791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.646666 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.646791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.646717 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.646968 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.646801 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.646968 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.646852 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.646968 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.646884 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.647138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.646998 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wm8r\" (UniqueName: \"kubernetes.io/projected/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-kube-api-access-9wm8r\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.647138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.647039 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.647287 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.647147 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.647405 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.647381 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.647528 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.647461 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.648913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.648891 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.649152 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.649132 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.671828 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.671792 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.671989 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.671898 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wm8r\" (UniqueName: \"kubernetes.io/projected/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-kube-api-access-9wm8r\") pod \"maas-default-gateway-openshift-default-55fbd65575-7vb5k\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:39.688048 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:39.688019 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:40.019544 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:40.019511 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k"] Oct 29 23:26:40.024178 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:26:40.024148 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df5f43c_d322_4e8a_bb0f_ae13dc90e54f.slice/crio-3da8692efc4df9fee4b445de0b7a3d234feaac97db6989fdc3d020445e8266fa WatchSource:0}: Error finding container 3da8692efc4df9fee4b445de0b7a3d234feaac97db6989fdc3d020445e8266fa: Status 404 returned error can't find the container with id 3da8692efc4df9fee4b445de0b7a3d234feaac97db6989fdc3d020445e8266fa Oct 29 23:26:40.027664 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:40.027626 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:26:40.027774 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:40.027704 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:26:40.027774 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:40.027744 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:26:40.815274 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:40.815235 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" event={"ID":"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f","Type":"ContainerStarted","Data":"b2710aba657110990e9a1a8198fe8c23e45357fad2df5697e0026562d17a8a36"} Oct 29 23:26:40.815795 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:40.815305 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" event={"ID":"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f","Type":"ContainerStarted","Data":"3da8692efc4df9fee4b445de0b7a3d234feaac97db6989fdc3d020445e8266fa"} Oct 29 23:26:40.878532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:40.878480 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" podStartSLOduration=1.87846404 podStartE2EDuration="1.87846404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:40.876664924 +0000 UTC m=+2647.528163453" watchObservedRunningTime="2026-10-29 23:26:40.87846404 +0000 UTC m=+2647.529962571" Oct 29 23:26:41.688766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:41.688724 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:41.691271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:41.691229 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-7vb5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 29 23:26:41.691450 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:41.691297 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" podUID="3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 29 23:26:42.688590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:42.688552 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-7vb5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 29 23:26:42.688978 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:42.688619 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" podUID="3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 29 23:26:43.688587 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:43.688552 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-7vb5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 29 23:26:43.688769 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:43.688606 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" podUID="3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 29 23:26:44.688853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:44.688810 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-7vb5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 29 23:26:44.689271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:44.688888 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" podUID="3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 29 23:26:45.688675 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:45.688633 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-7vb5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 29 23:26:45.688904 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:45.688686 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" podUID="3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 29 23:26:46.690481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:46.690396 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-7vb5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 29 23:26:46.690992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:46.690488 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" podUID="3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 29 23:26:47.688803 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:47.688761 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-7vb5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 29 23:26:47.689016 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:47.688841 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" podUID="3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 29 23:26:49.315493 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:49.315454 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:49.316062 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:49.316041 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:26:49.316996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:26:49.316974 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:27:03.998971 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:03.998934 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:27:33.914693 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:33.914654 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:27:35.039786 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:35.039758 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:27:35.058396 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:35.058368 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:27:39.575939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.575909 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4"] Oct 29 23:27:39.585956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.585934 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.599577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.599547 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml"] Oct 29 23:27:39.605710 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.605684 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.615338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.615310 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4"] Oct 29 23:27:39.631917 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.631665 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44918f77-9e08-4a3e-ba96-95763437e8ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.632807 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.631810 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.637088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.637061 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.637432 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.637409 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.637602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.637583 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.637794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.637775 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.638015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.637997 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.638146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.638130 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcsg2\" (UniqueName: \"kubernetes.io/projected/44918f77-9e08-4a3e-ba96-95763437e8ea-kube-api-access-zcsg2\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.638352 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.638334 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.648175 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.647584 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml"] Oct 29 23:27:39.740534 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.740499 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.740709 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.740538 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44918f77-9e08-4a3e-ba96-95763437e8ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.740709 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.740640 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.740709 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.740674 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.740818 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.740774 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.740818 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.740809 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.740939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.740851 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.740939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.740923 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qshx8\" (UniqueName: \"kubernetes.io/projected/a846f9dc-451b-48f6-b370-cdd5276667ed-kube-api-access-qshx8\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.741077 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.741060 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.741143 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.741114 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.741223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.741148 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a846f9dc-451b-48f6-b370-cdd5276667ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.741223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.741214 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.741344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.741242 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.741344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.741260 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.741344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.741306 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcsg2\" (UniqueName: \"kubernetes.io/projected/44918f77-9e08-4a3e-ba96-95763437e8ea-kube-api-access-zcsg2\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.741539 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.741420 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44918f77-9e08-4a3e-ba96-95763437e8ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.741539 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.741513 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.741539 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.741524 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.741723 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.741544 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.741723 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.741595 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.741723 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.741693 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.741723 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.741729 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.742032 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.741919 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.743543 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.743514 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.743787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.743765 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.772175 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.772144 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.772371 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.772348 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcsg2\" (UniqueName: \"kubernetes.io/projected/44918f77-9e08-4a3e-ba96-95763437e8ea-kube-api-access-zcsg2\") pod \"maas-default-gateway-openshift-default-55fbd65575-58ks4\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.843129 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.843044 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qshx8\" (UniqueName: \"kubernetes.io/projected/a846f9dc-451b-48f6-b370-cdd5276667ed-kube-api-access-qshx8\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.843317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.843146 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a846f9dc-451b-48f6-b370-cdd5276667ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.843317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.843268 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.843444 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.843319 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.843444 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.843366 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.843562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.843469 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.843562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.843542 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.843686 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.843577 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.843686 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.843674 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.843797 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.843682 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.843797 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.843750 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.844049 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.844028 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a846f9dc-451b-48f6-b370-cdd5276667ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.844131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.844036 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.844131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.844090 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.846104 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.846078 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.846547 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.846529 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.868512 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.868486 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.868682 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.868543 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qshx8\" (UniqueName: \"kubernetes.io/projected/a846f9dc-451b-48f6-b370-cdd5276667ed-kube-api-access-qshx8\") pod \"maas-default-gateway-openshift-default-55fbd65575-c5jml\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:39.923157 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.923127 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:39.947263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:39.947176 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:40.338853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:40.338812 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:27:40.339013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:40.338897 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:27:40.339013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:40.338933 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:27:40.340436 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:40.340408 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4"] Oct 29 23:27:40.350111 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:40.350088 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml"] Oct 29 23:27:40.350650 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:27:40.350613 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda846f9dc_451b_48f6_b370_cdd5276667ed.slice/crio-25e4dc835f183e8285b6ebba02c04100d441df04bbd495ea0a6d123ee432a4f5 WatchSource:0}: Error finding container 25e4dc835f183e8285b6ebba02c04100d441df04bbd495ea0a6d123ee432a4f5: Status 404 returned error can't find the container with id 25e4dc835f183e8285b6ebba02c04100d441df04bbd495ea0a6d123ee432a4f5 Oct 29 23:27:40.355142 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:40.355100 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:27:40.355296 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:40.355180 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:27:40.355296 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:40.355243 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:27:40.914403 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:40.914367 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" event={"ID":"44918f77-9e08-4a3e-ba96-95763437e8ea","Type":"ContainerStarted","Data":"04be180318370ab58a2bcf850febb64bfc735bac26cbc8e27a599c3e1df678e7"} Oct 29 23:27:40.914899 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:40.914425 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" event={"ID":"44918f77-9e08-4a3e-ba96-95763437e8ea","Type":"ContainerStarted","Data":"7c7dbb04d1f86a090188c5025cef4dbfce142aae35a2830b817fe9c70a2772c4"} Oct 29 23:27:40.916510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:40.916472 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" event={"ID":"a846f9dc-451b-48f6-b370-cdd5276667ed","Type":"ContainerStarted","Data":"811248c3b1cbbc27cb71f15927e6f256d192b6ee4bfa502a7875b44f8b97165b"} Oct 29 23:27:40.916510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:40.916539 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" event={"ID":"a846f9dc-451b-48f6-b370-cdd5276667ed","Type":"ContainerStarted","Data":"25e4dc835f183e8285b6ebba02c04100d441df04bbd495ea0a6d123ee432a4f5"} Oct 29 23:27:40.923821 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:40.923761 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:40.959212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:40.947441 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:40.979485 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:40.979422 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" podStartSLOduration=1.97940257 podStartE2EDuration="1.97940257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:40.97626246 +0000 UTC m=+2707.627760991" watchObservedRunningTime="2026-10-29 23:27:40.97940257 +0000 UTC m=+2707.630901101" Oct 29 23:27:41.040487 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:41.040421 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" podStartSLOduration=2.040398616 podStartE2EDuration="2.040398616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:41.024028157 +0000 UTC m=+2707.675526688" watchObservedRunningTime="2026-10-29 23:27:41.040398616 +0000 UTC m=+2707.691897146" Oct 29 23:27:41.803220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:41.797108 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6"] Oct 29 23:27:42.015712 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:42.003800 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-58ks4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 29 23:27:42.015712 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:42.003860 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" podUID="44918f77-9e08-4a3e-ba96-95763437e8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 29 23:27:42.015712 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:42.004785 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-c5jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 29 23:27:42.015712 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:42.004865 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" podUID="a846f9dc-451b-48f6-b370-cdd5276667ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 29 23:27:42.932208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:42.928756 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-58ks4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 29 23:27:42.932208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:42.928835 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" podUID="44918f77-9e08-4a3e-ba96-95763437e8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 29 23:27:42.956538 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:42.956490 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-c5jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 29 23:27:42.956833 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:42.956788 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" podUID="a846f9dc-451b-48f6-b370-cdd5276667ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 29 23:27:43.934208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:43.923524 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-58ks4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 29 23:27:43.934208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:43.923586 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" podUID="44918f77-9e08-4a3e-ba96-95763437e8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 29 23:27:43.965213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:43.957846 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-c5jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 29 23:27:43.965213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:43.957909 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" podUID="a846f9dc-451b-48f6-b370-cdd5276667ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 29 23:27:44.925213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:44.924702 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-58ks4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 29 23:27:44.925213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:44.924764 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" podUID="44918f77-9e08-4a3e-ba96-95763437e8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 29 23:27:44.948663 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:44.948545 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-c5jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 29 23:27:44.948663 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:44.948606 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" podUID="a846f9dc-451b-48f6-b370-cdd5276667ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 29 23:27:45.420282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:45.419937 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n"] Oct 29 23:27:45.420471 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:45.420331 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" podUID="8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" containerName="istio-proxy" containerID="cri-o://bb48e2e1ff4fb066d0f52cbbc4ac90cd2fd3d6ee5ccbe70cfb19cb3eb3d3eb90" gracePeriod=30 Oct 29 23:27:45.929605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:45.929567 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-58ks4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 29 23:27:45.929923 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:45.929894 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" podUID="44918f77-9e08-4a3e-ba96-95763437e8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 29 23:27:45.947631 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:45.947596 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-c5jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 29 23:27:45.947901 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:45.947874 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" podUID="a846f9dc-451b-48f6-b370-cdd5276667ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 29 23:27:46.806293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:46.806244 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" podUID="7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0" containerName="payload-processing" containerID="cri-o://bb2358e307b54e12411cc8379929467c96da9d40b531d1afe7ba182a893a6d69" gracePeriod=25 Oct 29 23:27:46.949266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:46.944251 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-58ks4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 29 23:27:46.949266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:46.944326 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" podUID="44918f77-9e08-4a3e-ba96-95763437e8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 29 23:27:46.954212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:46.951275 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-c5jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 29 23:27:46.954212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:46.951342 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" podUID="a846f9dc-451b-48f6-b370-cdd5276667ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 29 23:27:47.676087 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:47.676058 5057 generic.go:356] "Generic (PLEG): container finished" podID="7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0" containerID="bb2358e307b54e12411cc8379929467c96da9d40b531d1afe7ba182a893a6d69" exitCode=0 Oct 29 23:27:47.676267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:47.676168 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" event={"ID":"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0","Type":"ContainerDied","Data":"bb2358e307b54e12411cc8379929467c96da9d40b531d1afe7ba182a893a6d69"} Oct 29 23:27:47.740212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:47.726561 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" Oct 29 23:27:47.815016 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:47.814979 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-plugins-config-volume\") pod \"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0\" (UID: \"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0\") " Oct 29 23:27:47.815584 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:47.815266 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-routing-overlay-volume\") pod \"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0\" (UID: \"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0\") " Oct 29 23:27:47.815584 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:47.815397 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-provider-credentials\") pod \"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0\" (UID: \"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0\") " Oct 29 23:27:47.815584 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:47.815472 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-kube-api-access-4n7jc\" (UniqueName: \"kubernetes.io/projected/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-kube-api-access-4n7jc\") pod \"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0\" (UID: \"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0\") " Oct 29 23:27:47.816231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:47.816203 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-routing-overlay-volume" pod "7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0" (UID: "7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:47.816829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:47.816788 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-plugins-config-volume" pod "7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0" (UID: "7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:47.819078 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:47.819055 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-provider-credentials" pod "7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0" (UID: "7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:47.819214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:47.819156 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-kube-api-access-4n7jc" pod "7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0" (UID: "7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0"). InnerVolumeSpecName "kube-api-access-4n7jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:47.918677 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:47.918632 5057 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-routing-overlay-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:27:47.918677 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:47.918677 5057 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-provider-credentials\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:27:47.918898 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:47.918692 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n7jc\" (UniqueName: \"kubernetes.io/projected/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-kube-api-access-4n7jc\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:27:47.918898 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:47.918704 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:27:47.964393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:47.964356 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-58ks4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 29 23:27:47.964665 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:47.964640 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" podUID="44918f77-9e08-4a3e-ba96-95763437e8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 29 23:27:47.965170 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:47.965145 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-c5jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 29 23:27:47.966237 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:47.966213 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" podUID="a846f9dc-451b-48f6-b370-cdd5276667ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 29 23:27:48.693310 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:48.690890 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" Oct 29 23:27:48.693310 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:48.691068 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6" event={"ID":"7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0","Type":"ContainerDied","Data":"26e4ef912dab9e8351f072276cfb4831c314ee6624e4f6df60756a4deafbac19"} Oct 29 23:27:48.693310 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:48.691133 5057 scope.go:122] "RemoveContainer" containerID="bb2358e307b54e12411cc8379929467c96da9d40b531d1afe7ba182a893a6d69" Oct 29 23:27:48.768567 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:48.768524 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6"] Oct 29 23:27:48.774518 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:48.774480 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-844cdcdcd7-jxft6"] Oct 29 23:27:48.932429 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:48.924872 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-58ks4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 29 23:27:48.932429 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:48.924947 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" podUID="44918f77-9e08-4a3e-ba96-95763437e8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 29 23:27:48.961259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:48.960763 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-c5jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 29 23:27:48.961259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:48.960822 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" podUID="a846f9dc-451b-48f6-b370-cdd5276667ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 29 23:27:49.546340 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:49.546295 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0" path="/var/lib/kubelet/pods/7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0/volumes" Oct 29 23:27:50.967777 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:50.961627 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-58ks4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:27:50.967777 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:50.961711 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" podUID="44918f77-9e08-4a3e-ba96-95763437e8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:27:50.988324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:50.978313 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-c5jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:27:50.988324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:50.978417 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" podUID="a846f9dc-451b-48f6-b370-cdd5276667ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:27:51.090182 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.080446 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:51.090182 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.084348 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:51.090182 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.086582 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:27:51.164453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.163728 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:27:51.185084 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.185050 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-kube-api-access-xpbsz\" (UniqueName: \"kubernetes.io/projected/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-kube-api-access-xpbsz\") pod \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " Oct 29 23:27:51.185302 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.185097 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-data\") pod \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " Oct 29 23:27:51.185302 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.185245 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-envoy\") pod \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " Oct 29 23:27:51.185302 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.185281 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-workload-certs\") pod \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " Oct 29 23:27:51.185474 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.185450 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-podinfo\") pod \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " Oct 29 23:27:51.185537 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.185527 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-token\" (UniqueName: \"kubernetes.io/projected/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-token\") pod \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " Oct 29 23:27:51.185704 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.185668 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-credential-socket\") pod \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " Oct 29 23:27:51.185704 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.185671 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-workload-certs" pod "8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" (UID: "8cb171dd-4b28-4381-b1a5-7e5a25f2aefd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:51.185846 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.185738 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istiod-ca-cert\") pod \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " Oct 29 23:27:51.185846 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.185776 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-workload-socket\") pod \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\" (UID: \"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd\") " Oct 29 23:27:51.186355 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.186324 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istiod-ca-cert" pod "8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" (UID: "8cb171dd-4b28-4381-b1a5-7e5a25f2aefd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:51.186494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.186472 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-data" pod "8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" (UID: "8cb171dd-4b28-4381-b1a5-7e5a25f2aefd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:51.186571 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.186523 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-credential-socket" pod "8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" (UID: "8cb171dd-4b28-4381-b1a5-7e5a25f2aefd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:51.186691 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.186668 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-workload-socket" pod "8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" (UID: "8cb171dd-4b28-4381-b1a5-7e5a25f2aefd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:51.187263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.187155 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:27:51.187263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.187182 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:27:51.187263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.187216 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:27:51.187263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.187231 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:27:51.187263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.187243 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:27:51.189501 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.189473 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-token" pod "8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" (UID: "8cb171dd-4b28-4381-b1a5-7e5a25f2aefd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:51.189936 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.189884 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-podinfo" pod "8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" (UID: "8cb171dd-4b28-4381-b1a5-7e5a25f2aefd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:27:51.190310 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.190248 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-kube-api-access-xpbsz" pod "8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" (UID: "8cb171dd-4b28-4381-b1a5-7e5a25f2aefd"). InnerVolumeSpecName "kube-api-access-xpbsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:51.190517 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.190483 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-envoy" pod "8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" (UID: "8cb171dd-4b28-4381-b1a5-7e5a25f2aefd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:51.289202 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.289108 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:27:51.289375 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.289363 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:27:51.289450 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.289443 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:27:51.289512 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.289503 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpbsz\" (UniqueName: \"kubernetes.io/projected/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd-kube-api-access-xpbsz\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:27:51.733212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.732298 5057 generic.go:356] "Generic (PLEG): container finished" podID="8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" containerID="bb48e2e1ff4fb066d0f52cbbc4ac90cd2fd3d6ee5ccbe70cfb19cb3eb3d3eb90" exitCode=0 Oct 29 23:27:51.733438 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.733298 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" Oct 29 23:27:51.737206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.734332 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" event={"ID":"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd","Type":"ContainerDied","Data":"bb48e2e1ff4fb066d0f52cbbc4ac90cd2fd3d6ee5ccbe70cfb19cb3eb3d3eb90"} Oct 29 23:27:51.737206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.734396 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n" event={"ID":"8cb171dd-4b28-4381-b1a5-7e5a25f2aefd","Type":"ContainerDied","Data":"1e6c57a923cdfc34e5472bebe5a86d19ae40dfee51596f99a3666bdf9e49252e"} Oct 29 23:27:51.737206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.734427 5057 scope.go:122] "RemoveContainer" containerID="bb48e2e1ff4fb066d0f52cbbc4ac90cd2fd3d6ee5ccbe70cfb19cb3eb3d3eb90" Oct 29 23:27:51.756155 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.756081 5057 scope.go:122] "RemoveContainer" containerID="bb48e2e1ff4fb066d0f52cbbc4ac90cd2fd3d6ee5ccbe70cfb19cb3eb3d3eb90" Oct 29 23:27:51.756811 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:51.756777 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb48e2e1ff4fb066d0f52cbbc4ac90cd2fd3d6ee5ccbe70cfb19cb3eb3d3eb90\": container with ID starting with bb48e2e1ff4fb066d0f52cbbc4ac90cd2fd3d6ee5ccbe70cfb19cb3eb3d3eb90 not found: ID does not exist" containerID="bb48e2e1ff4fb066d0f52cbbc4ac90cd2fd3d6ee5ccbe70cfb19cb3eb3d3eb90" Oct 29 23:27:51.756973 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.756947 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb48e2e1ff4fb066d0f52cbbc4ac90cd2fd3d6ee5ccbe70cfb19cb3eb3d3eb90"} err="failed to get container status \"bb48e2e1ff4fb066d0f52cbbc4ac90cd2fd3d6ee5ccbe70cfb19cb3eb3d3eb90\": rpc error: code = NotFound desc = could not find container \"bb48e2e1ff4fb066d0f52cbbc4ac90cd2fd3d6ee5ccbe70cfb19cb3eb3d3eb90\": container with ID starting with bb48e2e1ff4fb066d0f52cbbc4ac90cd2fd3d6ee5ccbe70cfb19cb3eb3d3eb90 not found: ID does not exist" Oct 29 23:27:51.798364 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.798311 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n"] Oct 29 23:27:51.814951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.814909 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-bs87n"] Oct 29 23:27:51.979429 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.979303 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-c5jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:27:51.979429 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:51.979380 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" podUID="a846f9dc-451b-48f6-b370-cdd5276667ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:27:53.045233 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:53.023297 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-c5jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:27:53.045233 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:53.023365 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" podUID="a846f9dc-451b-48f6-b370-cdd5276667ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:27:53.506891 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:53.506859 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" path="/var/lib/kubelet/pods/8cb171dd-4b28-4381-b1a5-7e5a25f2aefd/volumes" Oct 29 23:27:54.089851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:54.086522 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:27:54.089851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:54.086591 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" podUID="c6508b22-be11-4d60-80bd-42a88a299853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:27:54.089851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:54.088224 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-c5jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:27:54.089851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:54.088272 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" podUID="a846f9dc-451b-48f6-b370-cdd5276667ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:27:54.130632 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:54.130492 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:54.132473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:54.132273 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:54.138415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:54.138389 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:27:55.613065 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.610889 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w"] Oct 29 23:27:55.613065 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.612580 5057 cpu_manager.go:414] "containerMap: removing container" podUID="8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" containerName="istio-proxy" Oct 29 23:27:55.613065 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.612602 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" containerName="istio-proxy" Oct 29 23:27:55.613065 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.612632 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0" containerName="payload-processing" Oct 29 23:27:55.613065 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.612640 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0" containerName="payload-processing" Oct 29 23:27:55.613065 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.612914 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7660eca1-9fd8-4b74-9fb1-f3ce4d5c16a0" containerName="payload-processing" Oct 29 23:27:55.613065 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.612948 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb171dd-4b28-4381-b1a5-7e5a25f2aefd" containerName="istio-proxy" Oct 29 23:27:55.705230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.680954 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w"] Oct 29 23:27:55.705230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.681071 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" Oct 29 23:27:55.706071 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.705572 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ee0a603072ca\"" Oct 29 23:27:55.706071 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.705678 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 29 23:27:55.706071 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.705819 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 29 23:27:55.706071 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.705939 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-6drnx\"" Oct 29 23:27:55.706071 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.706000 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 29 23:27:55.707937 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.706641 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 29 23:27:55.753289 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.747695 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf2d7\" (UniqueName: \"kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-kube-api-access-xf2d7\") pod \"payload-processing-external-model-7c5dfb7b64-x842w\" (UID: \"cc0d017d-60ca-4014-876a-419296cd9f35\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" Oct 29 23:27:55.753289 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.747894 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc0d017d-60ca-4014-876a-419296cd9f35-routing-overlay-volume\") pod \"payload-processing-external-model-7c5dfb7b64-x842w\" (UID: \"cc0d017d-60ca-4014-876a-419296cd9f35\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" Oct 29 23:27:55.753289 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.747987 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-provider-credentials\") pod \"payload-processing-external-model-7c5dfb7b64-x842w\" (UID: \"cc0d017d-60ca-4014-876a-419296cd9f35\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" Oct 29 23:27:55.753289 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.748095 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0d017d-60ca-4014-876a-419296cd9f35-plugins-config-volume\") pod \"payload-processing-external-model-7c5dfb7b64-x842w\" (UID: \"cc0d017d-60ca-4014-876a-419296cd9f35\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" Oct 29 23:27:55.954239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.942843 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc0d017d-60ca-4014-876a-419296cd9f35-routing-overlay-volume\") pod \"payload-processing-external-model-7c5dfb7b64-x842w\" (UID: \"cc0d017d-60ca-4014-876a-419296cd9f35\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" Oct 29 23:27:55.954239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.942940 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-provider-credentials\") pod \"payload-processing-external-model-7c5dfb7b64-x842w\" (UID: \"cc0d017d-60ca-4014-876a-419296cd9f35\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" Oct 29 23:27:55.954239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.943039 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0d017d-60ca-4014-876a-419296cd9f35-plugins-config-volume\") pod \"payload-processing-external-model-7c5dfb7b64-x842w\" (UID: \"cc0d017d-60ca-4014-876a-419296cd9f35\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" Oct 29 23:27:55.954239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.943347 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf2d7\" (UniqueName: \"kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-kube-api-access-xf2d7\") pod \"payload-processing-external-model-7c5dfb7b64-x842w\" (UID: \"cc0d017d-60ca-4014-876a-419296cd9f35\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" Oct 29 23:27:55.954239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:55.944648 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc0d017d-60ca-4014-876a-419296cd9f35-routing-overlay-volume\") pod \"payload-processing-external-model-7c5dfb7b64-x842w\" (UID: \"cc0d017d-60ca-4014-876a-419296cd9f35\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" Oct 29 23:27:55.954239 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:55.944761 5057 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ee0a603072ca: secret "agc-cred-ee0a603072ca" not found Oct 29 23:27:55.954239 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:55.944776 5057 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w: secret "agc-cred-ee0a603072ca" not found Oct 29 23:27:55.954239 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:55.944828 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-provider-credentials podName:cc0d017d-60ca-4014-876a-419296cd9f35 nodeName:}" failed. No retries permitted until 2026-10-29 23:27:56.444809377 +0000 UTC m=+2723.096307903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-provider-credentials") pod "payload-processing-external-model-7c5dfb7b64-x842w" (UID: "cc0d017d-60ca-4014-876a-419296cd9f35") : secret "agc-cred-ee0a603072ca" not found Oct 29 23:27:55.954239 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:55.945040 5057 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 29 23:27:55.954239 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:55.945080 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc0d017d-60ca-4014-876a-419296cd9f35-plugins-config-volume podName:cc0d017d-60ca-4014-876a-419296cd9f35 nodeName:}" failed. No retries permitted until 2026-10-29 23:27:56.445067495 +0000 UTC m=+2723.096566019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cc0d017d-60ca-4014-876a-419296cd9f35-plugins-config-volume") pod "payload-processing-external-model-7c5dfb7b64-x842w" (UID: "cc0d017d-60ca-4014-876a-419296cd9f35") : configmap "payload-processing-external-model-plugins" not found Oct 29 23:27:56.042215 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:56.024460 5057 projected.go:196] Error preparing data for projected volume kube-api-access-xf2d7 for pod models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w: failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 29 23:27:56.042215 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:56.024541 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-kube-api-access-xf2d7 podName:cc0d017d-60ca-4014-876a-419296cd9f35 nodeName:}" failed. No retries permitted until 2026-10-29 23:27:56.524519039 +0000 UTC m=+2723.176017547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xf2d7" (UniqueName: "kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-kube-api-access-xf2d7") pod "payload-processing-external-model-7c5dfb7b64-x842w" (UID: "cc0d017d-60ca-4014-876a-419296cd9f35") : failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 29 23:27:56.111213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:56.095697 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w"] Oct 29 23:27:56.111213 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:56.096391 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xf2d7 plugins-config-volume provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" podUID="cc0d017d-60ca-4014-876a-419296cd9f35" Oct 29 23:27:56.547209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:56.541204 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-provider-credentials\") pod \"payload-processing-external-model-7c5dfb7b64-x842w\" (UID: \"cc0d017d-60ca-4014-876a-419296cd9f35\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" Oct 29 23:27:56.547209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:56.541348 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0d017d-60ca-4014-876a-419296cd9f35-plugins-config-volume\") pod \"payload-processing-external-model-7c5dfb7b64-x842w\" (UID: \"cc0d017d-60ca-4014-876a-419296cd9f35\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" Oct 29 23:27:56.547209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:56.541642 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf2d7\" (UniqueName: \"kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-kube-api-access-xf2d7\") pod \"payload-processing-external-model-7c5dfb7b64-x842w\" (UID: \"cc0d017d-60ca-4014-876a-419296cd9f35\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" Oct 29 23:27:56.547209 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:56.542239 5057 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ee0a603072ca: secret "agc-cred-ee0a603072ca" not found Oct 29 23:27:56.547209 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:56.542258 5057 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w: secret "agc-cred-ee0a603072ca" not found Oct 29 23:27:56.547209 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:56.542310 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-provider-credentials podName:cc0d017d-60ca-4014-876a-419296cd9f35 nodeName:}" failed. No retries permitted until 2026-10-29 23:27:57.542290897 +0000 UTC m=+2724.193789405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-provider-credentials") pod "payload-processing-external-model-7c5dfb7b64-x842w" (UID: "cc0d017d-60ca-4014-876a-419296cd9f35") : secret "agc-cred-ee0a603072ca" not found Oct 29 23:27:56.547209 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:56.543081 5057 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 29 23:27:56.547209 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:56.543121 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc0d017d-60ca-4014-876a-419296cd9f35-plugins-config-volume podName:cc0d017d-60ca-4014-876a-419296cd9f35 nodeName:}" failed. No retries permitted until 2026-10-29 23:27:57.543109562 +0000 UTC m=+2724.194608082 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cc0d017d-60ca-4014-876a-419296cd9f35-plugins-config-volume") pod "payload-processing-external-model-7c5dfb7b64-x842w" (UID: "cc0d017d-60ca-4014-876a-419296cd9f35") : configmap "payload-processing-external-model-plugins" not found Oct 29 23:27:56.581213 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:56.566533 5057 projected.go:196] Error preparing data for projected volume kube-api-access-xf2d7 for pod models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w: failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 29 23:27:56.623460 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:56.623424 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-kube-api-access-xf2d7 podName:cc0d017d-60ca-4014-876a-419296cd9f35 nodeName:}" failed. No retries permitted until 2026-10-29 23:27:57.623389612 +0000 UTC m=+2724.274888133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xf2d7" (UniqueName: "kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-kube-api-access-xf2d7") pod "payload-processing-external-model-7c5dfb7b64-x842w" (UID: "cc0d017d-60ca-4014-876a-419296cd9f35") : failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 29 23:27:57.044066 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:57.044027 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" Oct 29 23:27:57.058962 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:57.058816 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" Oct 29 23:27:57.243207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:57.230096 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc0d017d-60ca-4014-876a-419296cd9f35-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc0d017d-60ca-4014-876a-419296cd9f35-routing-overlay-volume\") pod \"cc0d017d-60ca-4014-876a-419296cd9f35\" (UID: \"cc0d017d-60ca-4014-876a-419296cd9f35\") " Oct 29 23:27:57.243207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:57.231425 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0d017d-60ca-4014-876a-419296cd9f35-routing-overlay-volume" pod "cc0d017d-60ca-4014-876a-419296cd9f35" (UID: "cc0d017d-60ca-4014-876a-419296cd9f35"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:57.332916 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:57.332875 5057 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc0d017d-60ca-4014-876a-419296cd9f35-routing-overlay-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:27:57.637857 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:57.637767 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-provider-credentials\") pod \"payload-processing-external-model-7c5dfb7b64-x842w\" (UID: \"cc0d017d-60ca-4014-876a-419296cd9f35\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" Oct 29 23:27:57.638367 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:57.637869 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0d017d-60ca-4014-876a-419296cd9f35-plugins-config-volume\") pod \"payload-processing-external-model-7c5dfb7b64-x842w\" (UID: \"cc0d017d-60ca-4014-876a-419296cd9f35\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" Oct 29 23:27:57.638367 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:57.637950 5057 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 29 23:27:57.638367 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:57.638031 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc0d017d-60ca-4014-876a-419296cd9f35-plugins-config-volume podName:cc0d017d-60ca-4014-876a-419296cd9f35 nodeName:}" failed. No retries permitted until 2026-10-29 23:27:59.638009472 +0000 UTC m=+2726.289507996 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cc0d017d-60ca-4014-876a-419296cd9f35-plugins-config-volume") pod "payload-processing-external-model-7c5dfb7b64-x842w" (UID: "cc0d017d-60ca-4014-876a-419296cd9f35") : configmap "payload-processing-external-model-plugins" not found Oct 29 23:27:57.638367 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:57.638319 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf2d7\" (UniqueName: \"kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-kube-api-access-xf2d7\") pod \"payload-processing-external-model-7c5dfb7b64-x842w\" (UID: \"cc0d017d-60ca-4014-876a-419296cd9f35\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" Oct 29 23:27:57.640704 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:57.640677 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-provider-credentials\") pod \"payload-processing-external-model-7c5dfb7b64-x842w\" (UID: \"cc0d017d-60ca-4014-876a-419296cd9f35\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" Oct 29 23:27:57.655296 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:57.655262 5057 projected.go:196] Error preparing data for projected volume kube-api-access-xf2d7 for pod models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w: failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 29 23:27:57.656278 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:27:57.656249 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-kube-api-access-xf2d7 podName:cc0d017d-60ca-4014-876a-419296cd9f35 nodeName:}" failed. No retries permitted until 2026-10-29 23:27:59.655311635 +0000 UTC m=+2726.306810148 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xf2d7" (UniqueName: "kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-kube-api-access-xf2d7") pod "payload-processing-external-model-7c5dfb7b64-x842w" (UID: "cc0d017d-60ca-4014-876a-419296cd9f35") : failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 29 23:27:57.740072 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:57.740033 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-provider-credentials\") pod \"cc0d017d-60ca-4014-876a-419296cd9f35\" (UID: \"cc0d017d-60ca-4014-876a-419296cd9f35\") " Oct 29 23:27:57.742231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:57.742205 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-provider-credentials" pod "cc0d017d-60ca-4014-876a-419296cd9f35" (UID: "cc0d017d-60ca-4014-876a-419296cd9f35"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:57.868286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:57.861634 5057 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-provider-credentials\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:27:58.047991 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.043714 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65"] Oct 29 23:27:58.077746 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.072474 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" Oct 29 23:27:58.094277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.089061 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65"] Oct 29 23:27:58.097598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.096924 5057 generic.go:356] "Generic (PLEG): container finished" podID="c6508b22-be11-4d60-80bd-42a88a299853" containerID="2d557413580e7e4deb2404281cec8352a50820c901754479b16340047f6134fc" exitCode=0 Oct 29 23:27:58.097598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.097063 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w" Oct 29 23:27:58.099961 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.098086 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" event={"ID":"c6508b22-be11-4d60-80bd-42a88a299853","Type":"ContainerDied","Data":"2d557413580e7e4deb2404281cec8352a50820c901754479b16340047f6134fc"} Oct 29 23:27:58.100545 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.100523 5057 scope.go:122] "RemoveContainer" containerID="2d557413580e7e4deb2404281cec8352a50820c901754479b16340047f6134fc" Oct 29 23:27:58.103519 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.103490 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:27:58.103592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.103567 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:27:58.103638 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.103600 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:27:58.104021 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.103998 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-2crkv\"" Oct 29 23:27:58.263428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.189982 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a08e600-0dd8-4ea8-b116-e0742a5111f7-plugins-config-volume\") pod \"payload-processing-external-model-7c5dfb7b64-r4n65\" (UID: \"6a08e600-0dd8-4ea8-b116-e0742a5111f7\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" Oct 29 23:27:58.263428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.190162 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a08e600-0dd8-4ea8-b116-e0742a5111f7-provider-credentials\") pod \"payload-processing-external-model-7c5dfb7b64-r4n65\" (UID: \"6a08e600-0dd8-4ea8-b116-e0742a5111f7\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" Oct 29 23:27:58.263428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.190298 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5wt5\" (UniqueName: \"kubernetes.io/projected/6a08e600-0dd8-4ea8-b116-e0742a5111f7-kube-api-access-d5wt5\") pod \"payload-processing-external-model-7c5dfb7b64-r4n65\" (UID: \"6a08e600-0dd8-4ea8-b116-e0742a5111f7\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" Oct 29 23:27:58.263428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.190407 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a08e600-0dd8-4ea8-b116-e0742a5111f7-routing-overlay-volume\") pod \"payload-processing-external-model-7c5dfb7b64-r4n65\" (UID: \"6a08e600-0dd8-4ea8-b116-e0742a5111f7\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" Oct 29 23:27:58.447260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.412547 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a08e600-0dd8-4ea8-b116-e0742a5111f7-plugins-config-volume\") pod \"payload-processing-external-model-7c5dfb7b64-r4n65\" (UID: \"6a08e600-0dd8-4ea8-b116-e0742a5111f7\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" Oct 29 23:27:58.508592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.470562 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a08e600-0dd8-4ea8-b116-e0742a5111f7-plugins-config-volume\") pod \"payload-processing-external-model-7c5dfb7b64-r4n65\" (UID: \"6a08e600-0dd8-4ea8-b116-e0742a5111f7\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" Oct 29 23:27:58.508592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.471703 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a08e600-0dd8-4ea8-b116-e0742a5111f7-provider-credentials\") pod \"payload-processing-external-model-7c5dfb7b64-r4n65\" (UID: \"6a08e600-0dd8-4ea8-b116-e0742a5111f7\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" Oct 29 23:27:58.508592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.471806 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5wt5\" (UniqueName: \"kubernetes.io/projected/6a08e600-0dd8-4ea8-b116-e0742a5111f7-kube-api-access-d5wt5\") pod \"payload-processing-external-model-7c5dfb7b64-r4n65\" (UID: \"6a08e600-0dd8-4ea8-b116-e0742a5111f7\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" Oct 29 23:27:58.508592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.471904 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a08e600-0dd8-4ea8-b116-e0742a5111f7-routing-overlay-volume\") pod \"payload-processing-external-model-7c5dfb7b64-r4n65\" (UID: \"6a08e600-0dd8-4ea8-b116-e0742a5111f7\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" Oct 29 23:27:58.508592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.484006 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a08e600-0dd8-4ea8-b116-e0742a5111f7-routing-overlay-volume\") pod \"payload-processing-external-model-7c5dfb7b64-r4n65\" (UID: \"6a08e600-0dd8-4ea8-b116-e0742a5111f7\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" Oct 29 23:27:58.581283 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.580819 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a08e600-0dd8-4ea8-b116-e0742a5111f7-provider-credentials\") pod \"payload-processing-external-model-7c5dfb7b64-r4n65\" (UID: \"6a08e600-0dd8-4ea8-b116-e0742a5111f7\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" Oct 29 23:27:58.614332 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.602170 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5wt5\" (UniqueName: \"kubernetes.io/projected/6a08e600-0dd8-4ea8-b116-e0742a5111f7-kube-api-access-d5wt5\") pod \"payload-processing-external-model-7c5dfb7b64-r4n65\" (UID: \"6a08e600-0dd8-4ea8-b116-e0742a5111f7\") " pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" Oct 29 23:27:58.667071 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.667038 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w"] Oct 29 23:27:58.677532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.677458 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c5dfb7b64-x842w"] Oct 29 23:27:58.701771 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.701692 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" Oct 29 23:27:58.781387 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.781359 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0d017d-60ca-4014-876a-419296cd9f35-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:27:58.781387 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:58.781386 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf2d7\" (UniqueName: \"kubernetes.io/projected/cc0d017d-60ca-4014-876a-419296cd9f35-kube-api-access-xf2d7\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:27:59.029306 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:27:59.029164 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a08e600_0dd8_4ea8_b116_e0742a5111f7.slice/crio-b9cae4fdc04f18306def21ccba9432b6ff49951eaaaed9fd6137c7bece9472c7 WatchSource:0}: Error finding container b9cae4fdc04f18306def21ccba9432b6ff49951eaaaed9fd6137c7bece9472c7: Status 404 returned error can't find the container with id b9cae4fdc04f18306def21ccba9432b6ff49951eaaaed9fd6137c7bece9472c7 Oct 29 23:27:59.029980 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:59.029959 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65"] Oct 29 23:27:59.111288 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:59.111226 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" event={"ID":"6a08e600-0dd8-4ea8-b116-e0742a5111f7","Type":"ContainerStarted","Data":"b9cae4fdc04f18306def21ccba9432b6ff49951eaaaed9fd6137c7bece9472c7"} Oct 29 23:27:59.113878 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:59.113854 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" event={"ID":"c6508b22-be11-4d60-80bd-42a88a299853","Type":"ContainerStarted","Data":"b53553346862a1e1bc5277b978ef73b670bc1ae755b1ff086bbf45f663e785c5"} Oct 29 23:27:59.483766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:59.483732 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc0d017d-60ca-4014-876a-419296cd9f35" path="/var/lib/kubelet/pods/cc0d017d-60ca-4014-876a-419296cd9f35/volumes" Oct 29 23:27:59.978420 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:59.978370 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:27:59.979484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:59.979442 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 29 23:27:59.979678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:27:59.979494 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" podUID="c6508b22-be11-4d60-80bd-42a88a299853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 29 23:28:00.159399 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:00.159355 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" event={"ID":"6a08e600-0dd8-4ea8-b116-e0742a5111f7","Type":"ContainerStarted","Data":"6f6e2ce86b286e9f1f2be0274a562489e1cde98fb9969dee300f32aa6ce688b7"} Oct 29 23:28:00.159599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:00.159431 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" Oct 29 23:28:00.207308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:00.207257 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" podStartSLOduration=3.207242264 podStartE2EDuration="3.207242264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:00.204242211 +0000 UTC m=+2726.855740738" watchObservedRunningTime="2026-10-29 23:28:00.207242264 +0000 UTC m=+2726.858740793" Oct 29 23:28:00.979003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:00.978968 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 29 23:28:00.979552 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:00.979021 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" podUID="c6508b22-be11-4d60-80bd-42a88a299853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 29 23:28:01.978800 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:01.978764 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 29 23:28:01.978985 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:01.978828 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" podUID="c6508b22-be11-4d60-80bd-42a88a299853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 29 23:28:02.888948 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:02.888913 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65"] Oct 29 23:28:02.981737 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:02.981696 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 29 23:28:02.981934 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:02.981751 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" podUID="c6508b22-be11-4d60-80bd-42a88a299853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 29 23:28:03.917376 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:28:03.917335 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:28:03.978884 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:03.978848 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 29 23:28:03.979071 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:03.978902 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" podUID="c6508b22-be11-4d60-80bd-42a88a299853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 29 23:28:04.981496 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:04.978693 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 29 23:28:04.981496 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:04.978764 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" podUID="c6508b22-be11-4d60-80bd-42a88a299853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 29 23:28:05.981216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:05.981164 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 29 23:28:05.988444 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:05.988402 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" podUID="c6508b22-be11-4d60-80bd-42a88a299853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 29 23:28:07.063608 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:07.063555 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:28:07.065941 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:07.065916 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:28:07.076066 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:07.076040 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:28:07.843312 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:07.842444 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w"] Oct 29 23:28:07.859218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:07.857124 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" Oct 29 23:28:07.896213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:07.895517 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" podUID="6a08e600-0dd8-4ea8-b116-e0742a5111f7" containerName="payload-processing" containerID="cri-o://6f6e2ce86b286e9f1f2be0274a562489e1cde98fb9969dee300f32aa6ce688b7" gracePeriod=25 Oct 29 23:28:07.902214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:07.900835 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e1769a\"/\"agc-cred-0dd886d5c29d\"" Oct 29 23:28:07.911766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:07.910899 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w\" (UID: \"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" Oct 29 23:28:07.912341 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:07.912299 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98wvf\" (UniqueName: \"kubernetes.io/projected/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-kube-api-access-98wvf\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w\" (UID: \"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" Oct 29 23:28:07.912623 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:07.912559 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w\" (UID: \"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" Oct 29 23:28:07.912807 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:07.912737 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w\" (UID: \"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" Oct 29 23:28:07.944370 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:07.914603 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e1769a\"/\"kube-root-ca.crt\"" Oct 29 23:28:07.953208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:07.941364 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e1769a\"/\"openshift-service-ca.crt\"" Oct 29 23:28:07.953492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:07.941498 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e1769a\"/\"payload-processing-external-model-e2e--addccaec-dockercfg-2jmn5\"" Oct 29 23:28:07.983312 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:07.942378 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w"] Oct 29 23:28:08.183314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:07.983927 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e1769a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e1769a\"" Oct 29 23:28:08.183314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:08.009430 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e1769a\"/\"routing-overlay\"" Oct 29 23:28:08.183314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:08.133622 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w\" (UID: \"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" Oct 29 23:28:08.183314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:08.133706 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w\" (UID: \"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" Oct 29 23:28:08.183314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:08.179359 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w\" (UID: \"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" Oct 29 23:28:08.183314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:08.179778 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98wvf\" (UniqueName: \"kubernetes.io/projected/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-kube-api-access-98wvf\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w\" (UID: \"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" Oct 29 23:28:08.234245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:08.210978 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w\" (UID: \"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" Oct 29 23:28:08.286250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:08.256739 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98wvf\" (UniqueName: \"kubernetes.io/projected/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-kube-api-access-98wvf\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w\" (UID: \"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" Oct 29 23:28:08.340292 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:08.302664 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w\" (UID: \"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" Oct 29 23:28:08.344146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:08.343973 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w\" (UID: \"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" Oct 29 23:28:08.480889 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:08.476800 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" Oct 29 23:28:09.084449 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.046256 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5"] Oct 29 23:28:09.285290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.275621 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" Oct 29 23:28:09.285290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.277179 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5"] Oct 29 23:28:09.299151 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.299120 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-206095\"/\"agc-cred-40f288853eea\"" Oct 29 23:28:09.313135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.313068 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-206095\"/\"payload-processing-external-model-e2e--dc95ba64-dockercfg-7x46p\"" Oct 29 23:28:09.313361 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.313343 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-206095\"/\"kube-root-ca.crt\"" Oct 29 23:28:09.313455 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.313368 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-206095\"/\"routing-overlay\"" Oct 29 23:28:09.313455 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.313368 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-206095\"/\"openshift-service-ca.crt\"" Oct 29 23:28:09.313455 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.313377 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-206095\"/\"payload-processing-external-model-plugins-e2e-extmod-b-206095\"" Oct 29 23:28:09.437352 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.434556 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lln2\" (UniqueName: \"kubernetes.io/projected/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-kube-api-access-9lln2\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5\" (UID: \"81b0acc4-fa4c-4596-8a30-262c5c1d26ac\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" Oct 29 23:28:09.437352 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.434761 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5\" (UID: \"81b0acc4-fa4c-4596-8a30-262c5c1d26ac\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" Oct 29 23:28:09.437352 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.434911 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5\" (UID: \"81b0acc4-fa4c-4596-8a30-262c5c1d26ac\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" Oct 29 23:28:09.437352 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.435006 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5\" (UID: \"81b0acc4-fa4c-4596-8a30-262c5c1d26ac\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" Oct 29 23:28:09.458491 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.458217 5057 generic.go:356] "Generic (PLEG): container finished" podID="6a08e600-0dd8-4ea8-b116-e0742a5111f7" containerID="6f6e2ce86b286e9f1f2be0274a562489e1cde98fb9969dee300f32aa6ce688b7" exitCode=0 Oct 29 23:28:09.458491 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.458417 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" event={"ID":"6a08e600-0dd8-4ea8-b116-e0742a5111f7","Type":"ContainerDied","Data":"6f6e2ce86b286e9f1f2be0274a562489e1cde98fb9969dee300f32aa6ce688b7"} Oct 29 23:28:09.547877 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.547795 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5\" (UID: \"81b0acc4-fa4c-4596-8a30-262c5c1d26ac\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" Oct 29 23:28:09.548050 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.547948 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5\" (UID: \"81b0acc4-fa4c-4596-8a30-262c5c1d26ac\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" Oct 29 23:28:09.548050 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.548034 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5\" (UID: \"81b0acc4-fa4c-4596-8a30-262c5c1d26ac\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" Oct 29 23:28:09.548745 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.548710 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5\" (UID: \"81b0acc4-fa4c-4596-8a30-262c5c1d26ac\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" Oct 29 23:28:09.548874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.548773 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lln2\" (UniqueName: \"kubernetes.io/projected/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-kube-api-access-9lln2\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5\" (UID: \"81b0acc4-fa4c-4596-8a30-262c5c1d26ac\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" Oct 29 23:28:09.548943 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.548870 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5\" (UID: \"81b0acc4-fa4c-4596-8a30-262c5c1d26ac\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" Oct 29 23:28:09.551459 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.551428 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5\" (UID: \"81b0acc4-fa4c-4596-8a30-262c5c1d26ac\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" Oct 29 23:28:09.576975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.576949 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lln2\" (UniqueName: \"kubernetes.io/projected/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-kube-api-access-9lln2\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5\" (UID: \"81b0acc4-fa4c-4596-8a30-262c5c1d26ac\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" Oct 29 23:28:09.605268 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.605236 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" Oct 29 23:28:09.643580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.643544 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" Oct 29 23:28:09.692479 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.692203 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w"] Oct 29 23:28:09.752860 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.752830 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a08e600-0dd8-4ea8-b116-e0742a5111f7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a08e600-0dd8-4ea8-b116-e0742a5111f7-routing-overlay-volume\") pod \"6a08e600-0dd8-4ea8-b116-e0742a5111f7\" (UID: \"6a08e600-0dd8-4ea8-b116-e0742a5111f7\") " Oct 29 23:28:09.753340 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.753272 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a08e600-0dd8-4ea8-b116-e0742a5111f7-kube-api-access-d5wt5\" (UniqueName: \"kubernetes.io/projected/6a08e600-0dd8-4ea8-b116-e0742a5111f7-kube-api-access-d5wt5\") pod \"6a08e600-0dd8-4ea8-b116-e0742a5111f7\" (UID: \"6a08e600-0dd8-4ea8-b116-e0742a5111f7\") " Oct 29 23:28:09.753340 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.753302 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a08e600-0dd8-4ea8-b116-e0742a5111f7-routing-overlay-volume" pod "6a08e600-0dd8-4ea8-b116-e0742a5111f7" (UID: "6a08e600-0dd8-4ea8-b116-e0742a5111f7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:09.753340 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.753338 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a08e600-0dd8-4ea8-b116-e0742a5111f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a08e600-0dd8-4ea8-b116-e0742a5111f7-plugins-config-volume\") pod \"6a08e600-0dd8-4ea8-b116-e0742a5111f7\" (UID: \"6a08e600-0dd8-4ea8-b116-e0742a5111f7\") " Oct 29 23:28:09.753581 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.753418 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a08e600-0dd8-4ea8-b116-e0742a5111f7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a08e600-0dd8-4ea8-b116-e0742a5111f7-provider-credentials\") pod \"6a08e600-0dd8-4ea8-b116-e0742a5111f7\" (UID: \"6a08e600-0dd8-4ea8-b116-e0742a5111f7\") " Oct 29 23:28:09.753934 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.753740 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a08e600-0dd8-4ea8-b116-e0742a5111f7-plugins-config-volume" pod "6a08e600-0dd8-4ea8-b116-e0742a5111f7" (UID: "6a08e600-0dd8-4ea8-b116-e0742a5111f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:09.754371 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.754349 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a08e600-0dd8-4ea8-b116-e0742a5111f7-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:28:09.754446 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.754378 5057 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a08e600-0dd8-4ea8-b116-e0742a5111f7-routing-overlay-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:28:09.757437 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.757410 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a08e600-0dd8-4ea8-b116-e0742a5111f7-provider-credentials" pod "6a08e600-0dd8-4ea8-b116-e0742a5111f7" (UID: "6a08e600-0dd8-4ea8-b116-e0742a5111f7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:09.758081 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.758064 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a08e600-0dd8-4ea8-b116-e0742a5111f7-kube-api-access-d5wt5" pod "6a08e600-0dd8-4ea8-b116-e0742a5111f7" (UID: "6a08e600-0dd8-4ea8-b116-e0742a5111f7"). InnerVolumeSpecName "kube-api-access-d5wt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:09.856608 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.856516 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5wt5\" (UniqueName: \"kubernetes.io/projected/6a08e600-0dd8-4ea8-b116-e0742a5111f7-kube-api-access-d5wt5\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:28:09.856608 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.856555 5057 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a08e600-0dd8-4ea8-b116-e0742a5111f7-provider-credentials\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:28:09.919712 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:28:09.919684 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b0acc4_fa4c_4596_8a30_262c5c1d26ac.slice/crio-0e00e358c7a310ec4faee36bbb82b9b79d1dc0901873a2a026b8e757622a1da7 WatchSource:0}: Error finding container 0e00e358c7a310ec4faee36bbb82b9b79d1dc0901873a2a026b8e757622a1da7: Status 404 returned error can't find the container with id 0e00e358c7a310ec4faee36bbb82b9b79d1dc0901873a2a026b8e757622a1da7 Oct 29 23:28:09.920098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:09.920074 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5"] Oct 29 23:28:10.108627 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.108523 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv"] Oct 29 23:28:10.109812 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.109788 5057 cpu_manager.go:414] "containerMap: removing container" podUID="6a08e600-0dd8-4ea8-b116-e0742a5111f7" containerName="payload-processing" Oct 29 23:28:10.109812 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.109810 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a08e600-0dd8-4ea8-b116-e0742a5111f7" containerName="payload-processing" Oct 29 23:28:10.110068 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.110053 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a08e600-0dd8-4ea8-b116-e0742a5111f7" containerName="payload-processing" Oct 29 23:28:10.114381 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.114362 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.134526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.134483 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0e59cd-openshift-default-dockercfg-ssj9q\"" Oct 29 23:28:10.143035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.141923 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv"] Oct 29 23:28:10.163510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.163448 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-envoy\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.163851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.163828 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-credential-socket\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.164091 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.164073 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-data\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.164420 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.164388 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqgp8\" (UniqueName: \"kubernetes.io/projected/d032ce48-7955-47e5-a74b-9af1ef81ed17-kube-api-access-lqgp8\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.164643 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.164626 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-token\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.164843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.164827 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-workload-socket\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.165106 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.165079 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-workload-certs\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.165269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.165245 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-podinfo\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.165381 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.165286 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d032ce48-7955-47e5-a74b-9af1ef81ed17-istiod-ca-cert\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.275209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.267612 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-token\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.275209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.267760 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-workload-socket\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.275209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.267909 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-workload-certs\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.275209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.268017 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-podinfo\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.275209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.268047 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d032ce48-7955-47e5-a74b-9af1ef81ed17-istiod-ca-cert\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.275209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.268124 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-envoy\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.275209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.268153 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-credential-socket\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.279362 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.277664 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-workload-certs\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.285474 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.285438 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-data\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.304270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.304233 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqgp8\" (UniqueName: \"kubernetes.io/projected/d032ce48-7955-47e5-a74b-9af1ef81ed17-kube-api-access-lqgp8\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.304692 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.289611 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-workload-socket\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.304848 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.293524 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-credential-socket\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.304951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.293794 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-data\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.305051 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.289018 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d032ce48-7955-47e5-a74b-9af1ef81ed17-istiod-ca-cert\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.323548 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.323477 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-podinfo\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.327264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.327209 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-token\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.365260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.348572 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-envoy\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.366181 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.366116 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqgp8\" (UniqueName: \"kubernetes.io/projected/d032ce48-7955-47e5-a74b-9af1ef81ed17-kube-api-access-lqgp8\") pod \"e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.443765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.443489 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:10.523554 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.523508 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" event={"ID":"81b0acc4-fa4c-4596-8a30-262c5c1d26ac","Type":"ContainerStarted","Data":"a56994617f21205a173f1f7d580f4669e41c00afedec8c1f497b53f1d2cfeee0"} Oct 29 23:28:10.523847 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.523825 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" event={"ID":"81b0acc4-fa4c-4596-8a30-262c5c1d26ac","Type":"ContainerStarted","Data":"0e00e358c7a310ec4faee36bbb82b9b79d1dc0901873a2a026b8e757622a1da7"} Oct 29 23:28:10.523972 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.523960 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" Oct 29 23:28:10.533741 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.533710 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" event={"ID":"6a08e600-0dd8-4ea8-b116-e0742a5111f7","Type":"ContainerDied","Data":"b9cae4fdc04f18306def21ccba9432b6ff49951eaaaed9fd6137c7bece9472c7"} Oct 29 23:28:10.534045 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.533947 5057 scope.go:122] "RemoveContainer" containerID="6f6e2ce86b286e9f1f2be0274a562489e1cde98fb9969dee300f32aa6ce688b7" Oct 29 23:28:10.534405 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.534386 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65" Oct 29 23:28:10.597275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.588464 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" podStartSLOduration=2.588445079 podStartE2EDuration="2.588445079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:10.580445778 +0000 UTC m=+2737.231944311" watchObservedRunningTime="2026-10-29 23:28:10.588445079 +0000 UTC m=+2737.239943608" Oct 29 23:28:10.667676 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.667585 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" event={"ID":"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014","Type":"ContainerStarted","Data":"831df0393a302fe6a31e87d440e349d39d3537901b5b595ddfb1f4b1b1c2c4a1"} Oct 29 23:28:10.667946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.667913 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" event={"ID":"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014","Type":"ContainerStarted","Data":"e631ebec29b82cef6d962ac89aeef2792a0d43815ee8e0690a370bfd1bb06bdb"} Oct 29 23:28:10.669557 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.669527 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" Oct 29 23:28:10.811016 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.810920 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" podStartSLOduration=3.810895888 podStartE2EDuration="3.810895888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:10.721681301 +0000 UTC m=+2737.373179841" watchObservedRunningTime="2026-10-29 23:28:10.810895888 +0000 UTC m=+2737.462394419" Oct 29 23:28:10.821709 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.821672 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65"] Oct 29 23:28:10.829519 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:10.829485 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c5dfb7b64-r4n65"] Oct 29 23:28:11.244913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:11.244886 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv"] Oct 29 23:28:11.245347 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:28:11.245321 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd032ce48_7955_47e5_a74b_9af1ef81ed17.slice/crio-e6287e7461693a61f2b039fb5bd4044d8e006c0d2205feee91a263d039a3d796 WatchSource:0}: Error finding container e6287e7461693a61f2b039fb5bd4044d8e006c0d2205feee91a263d039a3d796: Status 404 returned error can't find the container with id e6287e7461693a61f2b039fb5bd4044d8e006c0d2205feee91a263d039a3d796 Oct 29 23:28:11.248329 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:11.248292 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:28:11.248451 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:11.248406 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:28:11.248451 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:11.248440 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:28:11.495036 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:11.494943 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a08e600-0dd8-4ea8-b116-e0742a5111f7" path="/var/lib/kubelet/pods/6a08e600-0dd8-4ea8-b116-e0742a5111f7/volumes" Oct 29 23:28:11.809723 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:11.809635 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" event={"ID":"d032ce48-7955-47e5-a74b-9af1ef81ed17","Type":"ContainerStarted","Data":"fd497a9ecdc6abe0f1405a9b283b31a198da24a6f58f8d93d1b99cb2d762063a"} Oct 29 23:28:11.809723 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:11.809690 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" event={"ID":"d032ce48-7955-47e5-a74b-9af1ef81ed17","Type":"ContainerStarted","Data":"e6287e7461693a61f2b039fb5bd4044d8e006c0d2205feee91a263d039a3d796"} Oct 29 23:28:11.866417 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:11.866358 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" podStartSLOduration=1.866338958 podStartE2EDuration="1.866338958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:11.861404448 +0000 UTC m=+2738.512902978" watchObservedRunningTime="2026-10-29 23:28:11.866338958 +0000 UTC m=+2738.517837490" Oct 29 23:28:12.444662 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:12.444628 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:12.451267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:12.451240 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:12.816494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:12.816410 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:12.818096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:12.818068 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:28:13.002123 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:13.002086 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:28:19.035389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:19.031594 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:28:19.050409 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:19.050369 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 23:28:21.813160 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:21.813130 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" Oct 29 23:28:22.819583 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:28:22.819547 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" Oct 29 23:28:33.915596 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:28:33.915555 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:29:03.915674 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:29:03.915628 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:29:11.828896 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:29:11.828835 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:29:12.079397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:29:12.079180 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 29 23:29:12.098644 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:29:12.096272 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:33.916395 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:29:33.916354 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:29:40.054339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:29:40.054303 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:30:00.181101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.181061 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs"] Oct 29 23:30:00.190748 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.190718 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf"] Oct 29 23:30:00.191370 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.191342 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" Oct 29 23:30:00.196812 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.196542 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" Oct 29 23:30:00.247085 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.247045 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs"] Oct 29 23:30:00.247489 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.247468 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd"] Oct 29 23:30:00.259350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.252591 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:30:00.296721 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.296681 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf"] Oct 29 23:30:00.298215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.298090 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd" Oct 29 23:30:00.298865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.298832 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z54mj\" (UniqueName: \"kubernetes.io/projected/8e2a04ca-f403-49e9-bd0c-f9f9394e188b-kube-api-access-z54mj\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf\" (UID: \"8e2a04ca-f403-49e9-bd0c-f9f9394e188b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" Oct 29 23:30:00.299372 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.298936 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2de74b6-1d73-4ab4-adaa-e081fbc0240d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs\" (UID: \"a2de74b6-1d73-4ab4-adaa-e081fbc0240d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" Oct 29 23:30:00.299372 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.299013 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2a04ca-f403-49e9-bd0c-f9f9394e188b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf\" (UID: \"8e2a04ca-f403-49e9-bd0c-f9f9394e188b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" Oct 29 23:30:00.299524 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.299362 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2425\" (UniqueName: \"kubernetes.io/projected/a2de74b6-1d73-4ab4-adaa-e081fbc0240d-kube-api-access-f2425\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs\" (UID: \"a2de74b6-1d73-4ab4-adaa-e081fbc0240d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" Oct 29 23:30:00.308449 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.308417 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z"] Oct 29 23:30:00.315169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.315133 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" Oct 29 23:30:00.317958 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.317904 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp"] Oct 29 23:30:00.365225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.364638 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" Oct 29 23:30:00.365408 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.365363 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd"] Oct 29 23:30:00.381152 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.380909 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6"] Oct 29 23:30:00.390659 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.390631 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6" Oct 29 23:30:00.401947 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.401910 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2a04ca-f403-49e9-bd0c-f9f9394e188b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf\" (UID: \"8e2a04ca-f403-49e9-bd0c-f9f9394e188b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" Oct 29 23:30:00.402126 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.402036 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sgb8\" (UniqueName: \"kubernetes.io/projected/eea3ec3f-906d-4185-b80b-00c0f50bee8e-kube-api-access-2sgb8\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd\" (UID: \"eea3ec3f-906d-4185-b80b-00c0f50bee8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd" Oct 29 23:30:00.402371 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.402346 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea3ec3f-906d-4185-b80b-00c0f50bee8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd\" (UID: \"eea3ec3f-906d-4185-b80b-00c0f50bee8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd" Oct 29 23:30:00.402472 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.402441 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2425\" (UniqueName: \"kubernetes.io/projected/a2de74b6-1d73-4ab4-adaa-e081fbc0240d-kube-api-access-f2425\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs\" (UID: \"a2de74b6-1d73-4ab4-adaa-e081fbc0240d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" Oct 29 23:30:00.402639 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.402605 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddknw\" (UniqueName: \"kubernetes.io/projected/9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8-kube-api-access-ddknw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z\" (UID: \"9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" Oct 29 23:30:00.402799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.402781 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z\" (UID: \"9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" Oct 29 23:30:00.402920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.402903 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z54mj\" (UniqueName: \"kubernetes.io/projected/8e2a04ca-f403-49e9-bd0c-f9f9394e188b-kube-api-access-z54mj\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf\" (UID: \"8e2a04ca-f403-49e9-bd0c-f9f9394e188b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" Oct 29 23:30:00.402992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.402973 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2de74b6-1d73-4ab4-adaa-e081fbc0240d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs\" (UID: \"a2de74b6-1d73-4ab4-adaa-e081fbc0240d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" Oct 29 23:30:00.404025 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.403998 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2de74b6-1d73-4ab4-adaa-e081fbc0240d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs\" (UID: \"a2de74b6-1d73-4ab4-adaa-e081fbc0240d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" Oct 29 23:30:00.404784 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.404760 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2a04ca-f403-49e9-bd0c-f9f9394e188b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf\" (UID: \"8e2a04ca-f403-49e9-bd0c-f9f9394e188b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" Oct 29 23:30:00.406321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.406300 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6"] Oct 29 23:30:00.417619 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.417587 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z"] Oct 29 23:30:00.417980 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.417957 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" Oct 29 23:30:00.435273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.435240 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh"] Oct 29 23:30:00.438868 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.438832 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2425\" (UniqueName: \"kubernetes.io/projected/a2de74b6-1d73-4ab4-adaa-e081fbc0240d-kube-api-access-f2425\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs\" (UID: \"a2de74b6-1d73-4ab4-adaa-e081fbc0240d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" Oct 29 23:30:00.448940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.448387 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6"] Oct 29 23:30:00.448940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.448530 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" Oct 29 23:30:00.455974 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.455945 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8"] Oct 29 23:30:00.456784 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.456760 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z54mj\" (UniqueName: \"kubernetes.io/projected/8e2a04ca-f403-49e9-bd0c-f9f9394e188b-kube-api-access-z54mj\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf\" (UID: \"8e2a04ca-f403-49e9-bd0c-f9f9394e188b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" Oct 29 23:30:00.468828 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.468798 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w"] Oct 29 23:30:00.475241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.469264 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8" Oct 29 23:30:00.477550 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.477438 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6"] Oct 29 23:30:00.477767 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.477527 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" Oct 29 23:30:00.486199 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.485510 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp"] Oct 29 23:30:00.500603 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.500477 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh"] Oct 29 23:30:00.505166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.505111 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sgb8\" (UniqueName: \"kubernetes.io/projected/eea3ec3f-906d-4185-b80b-00c0f50bee8e-kube-api-access-2sgb8\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd\" (UID: \"eea3ec3f-906d-4185-b80b-00c0f50bee8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd" Oct 29 23:30:00.505318 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.505231 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea3ec3f-906d-4185-b80b-00c0f50bee8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd\" (UID: \"eea3ec3f-906d-4185-b80b-00c0f50bee8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd" Oct 29 23:30:00.505318 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.505308 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stghw\" (UniqueName: \"kubernetes.io/projected/7eb091cb-8d53-4b37-835b-591edcba96f1-kube-api-access-stghw\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6\" (UID: \"7eb091cb-8d53-4b37-835b-591edcba96f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" Oct 29 23:30:00.505494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.505476 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddknw\" (UniqueName: \"kubernetes.io/projected/9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8-kube-api-access-ddknw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z\" (UID: \"9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" Oct 29 23:30:00.505561 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.505523 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/664c869e-4b17-41bb-946b-d2ead80eff08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp\" (UID: \"664c869e-4b17-41bb-946b-d2ead80eff08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" Oct 29 23:30:00.505623 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.505572 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb091cb-8d53-4b37-835b-591edcba96f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6\" (UID: \"7eb091cb-8d53-4b37-835b-591edcba96f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" Oct 29 23:30:00.505684 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.505632 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13000cb3-436f-4fec-bb70-7e09a449dec0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6\" (UID: \"13000cb3-436f-4fec-bb70-7e09a449dec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6" Oct 29 23:30:00.505740 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.505704 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z\" (UID: \"9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" Oct 29 23:30:00.505837 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.505822 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ctvb\" (UniqueName: \"kubernetes.io/projected/664c869e-4b17-41bb-946b-d2ead80eff08-kube-api-access-9ctvb\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp\" (UID: \"664c869e-4b17-41bb-946b-d2ead80eff08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" Oct 29 23:30:00.505897 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.505882 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp9p9\" (UniqueName: \"kubernetes.io/projected/13000cb3-436f-4fec-bb70-7e09a449dec0-kube-api-access-mp9p9\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6\" (UID: \"13000cb3-436f-4fec-bb70-7e09a449dec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6" Oct 29 23:30:00.506727 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.506700 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea3ec3f-906d-4185-b80b-00c0f50bee8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd\" (UID: \"eea3ec3f-906d-4185-b80b-00c0f50bee8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd" Oct 29 23:30:00.506831 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.506739 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z\" (UID: \"9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" Oct 29 23:30:00.509306 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.509275 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g"] Oct 29 23:30:00.514473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.514454 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g" Oct 29 23:30:00.516840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.516821 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w"] Oct 29 23:30:00.530836 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.530805 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddknw\" (UniqueName: \"kubernetes.io/projected/9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8-kube-api-access-ddknw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z\" (UID: \"9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" Oct 29 23:30:00.531236 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.531176 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8"] Oct 29 23:30:00.535374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.535347 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sgb8\" (UniqueName: \"kubernetes.io/projected/eea3ec3f-906d-4185-b80b-00c0f50bee8e-kube-api-access-2sgb8\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd\" (UID: \"eea3ec3f-906d-4185-b80b-00c0f50bee8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd" Oct 29 23:30:00.538910 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.538887 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv"] Oct 29 23:30:00.545018 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.544996 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv" Oct 29 23:30:00.547059 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.547040 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns"] Oct 29 23:30:00.553345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.553323 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns" Oct 29 23:30:00.564790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.564361 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv"] Oct 29 23:30:00.572750 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.572725 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g"] Oct 29 23:30:00.574858 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.574833 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" Oct 29 23:30:00.578475 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.578311 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns"] Oct 29 23:30:00.598295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.598267 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" Oct 29 23:30:00.607479 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.607449 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ctvb\" (UniqueName: \"kubernetes.io/projected/664c869e-4b17-41bb-946b-d2ead80eff08-kube-api-access-9ctvb\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp\" (UID: \"664c869e-4b17-41bb-946b-d2ead80eff08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" Oct 29 23:30:00.607972 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.607687 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp9p9\" (UniqueName: \"kubernetes.io/projected/13000cb3-436f-4fec-bb70-7e09a449dec0-kube-api-access-mp9p9\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6\" (UID: \"13000cb3-436f-4fec-bb70-7e09a449dec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6" Oct 29 23:30:00.607972 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.607750 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdv9m\" (UniqueName: \"kubernetes.io/projected/04c2e336-3da0-4816-8313-e8fdeb0edb5f-kube-api-access-wdv9m\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g\" (UID: \"04c2e336-3da0-4816-8313-e8fdeb0edb5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g" Oct 29 23:30:00.607972 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.607860 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f3e7bb-6ca1-46a8-a146-cb7c5690f377-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8\" (UID: \"e2f3e7bb-6ca1-46a8-a146-cb7c5690f377\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8" Oct 29 23:30:00.607972 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.607961 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04c2e336-3da0-4816-8313-e8fdeb0edb5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g\" (UID: \"04c2e336-3da0-4816-8313-e8fdeb0edb5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g" Oct 29 23:30:00.608581 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.608551 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stghw\" (UniqueName: \"kubernetes.io/projected/7eb091cb-8d53-4b37-835b-591edcba96f1-kube-api-access-stghw\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6\" (UID: \"7eb091cb-8d53-4b37-835b-591edcba96f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" Oct 29 23:30:00.608697 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.608652 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95wd9\" (UniqueName: \"kubernetes.io/projected/595f0d6b-00c5-4a95-8881-8191e7de38ff-kube-api-access-95wd9\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh\" (UID: \"595f0d6b-00c5-4a95-8881-8191e7de38ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" Oct 29 23:30:00.608770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.608695 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/595f0d6b-00c5-4a95-8881-8191e7de38ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh\" (UID: \"595f0d6b-00c5-4a95-8881-8191e7de38ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" Oct 29 23:30:00.608835 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.608770 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/664c869e-4b17-41bb-946b-d2ead80eff08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp\" (UID: \"664c869e-4b17-41bb-946b-d2ead80eff08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" Oct 29 23:30:00.608835 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.608821 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb091cb-8d53-4b37-835b-591edcba96f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6\" (UID: \"7eb091cb-8d53-4b37-835b-591edcba96f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" Oct 29 23:30:00.608948 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.608881 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13000cb3-436f-4fec-bb70-7e09a449dec0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6\" (UID: \"13000cb3-436f-4fec-bb70-7e09a449dec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6" Oct 29 23:30:00.608948 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.608906 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpdfx\" (UniqueName: \"kubernetes.io/projected/e2f3e7bb-6ca1-46a8-a146-cb7c5690f377-kube-api-access-cpdfx\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8\" (UID: \"e2f3e7bb-6ca1-46a8-a146-cb7c5690f377\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8" Oct 29 23:30:00.608948 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.608940 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2a4a82-5ab9-4c15-8a1d-54da076ff3db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w\" (UID: \"1c2a4a82-5ab9-4c15-8a1d-54da076ff3db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" Oct 29 23:30:00.609161 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.609043 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftkqt\" (UniqueName: \"kubernetes.io/projected/1c2a4a82-5ab9-4c15-8a1d-54da076ff3db-kube-api-access-ftkqt\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w\" (UID: \"1c2a4a82-5ab9-4c15-8a1d-54da076ff3db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" Oct 29 23:30:00.609930 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.609909 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb091cb-8d53-4b37-835b-591edcba96f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6\" (UID: \"7eb091cb-8d53-4b37-835b-591edcba96f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" Oct 29 23:30:00.610100 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.610074 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13000cb3-436f-4fec-bb70-7e09a449dec0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6\" (UID: \"13000cb3-436f-4fec-bb70-7e09a449dec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6" Oct 29 23:30:00.610855 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.610480 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/664c869e-4b17-41bb-946b-d2ead80eff08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp\" (UID: \"664c869e-4b17-41bb-946b-d2ead80eff08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" Oct 29 23:30:00.635250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.634968 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ctvb\" (UniqueName: \"kubernetes.io/projected/664c869e-4b17-41bb-946b-d2ead80eff08-kube-api-access-9ctvb\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp\" (UID: \"664c869e-4b17-41bb-946b-d2ead80eff08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" Oct 29 23:30:00.635250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.635165 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stghw\" (UniqueName: \"kubernetes.io/projected/7eb091cb-8d53-4b37-835b-591edcba96f1-kube-api-access-stghw\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6\" (UID: \"7eb091cb-8d53-4b37-835b-591edcba96f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" Oct 29 23:30:00.636161 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.636041 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp9p9\" (UniqueName: \"kubernetes.io/projected/13000cb3-436f-4fec-bb70-7e09a449dec0-kube-api-access-mp9p9\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6\" (UID: \"13000cb3-436f-4fec-bb70-7e09a449dec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6" Oct 29 23:30:00.699226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.671472 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" Oct 29 23:30:00.699226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.692486 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd" Oct 29 23:30:00.743213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.727500 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6" Oct 29 23:30:00.743213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.729850 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" Oct 29 23:30:00.743213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.735416 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95wd9\" (UniqueName: \"kubernetes.io/projected/595f0d6b-00c5-4a95-8881-8191e7de38ff-kube-api-access-95wd9\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh\" (UID: \"595f0d6b-00c5-4a95-8881-8191e7de38ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" Oct 29 23:30:00.752014 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.751935 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/595f0d6b-00c5-4a95-8881-8191e7de38ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh\" (UID: \"595f0d6b-00c5-4a95-8881-8191e7de38ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" Oct 29 23:30:00.752212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.752166 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpdfx\" (UniqueName: \"kubernetes.io/projected/e2f3e7bb-6ca1-46a8-a146-cb7c5690f377-kube-api-access-cpdfx\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8\" (UID: \"e2f3e7bb-6ca1-46a8-a146-cb7c5690f377\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8" Oct 29 23:30:00.754255 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.753467 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/595f0d6b-00c5-4a95-8881-8191e7de38ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh\" (UID: \"595f0d6b-00c5-4a95-8881-8191e7de38ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" Oct 29 23:30:00.757400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.755320 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2a4a82-5ab9-4c15-8a1d-54da076ff3db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w\" (UID: \"1c2a4a82-5ab9-4c15-8a1d-54da076ff3db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" Oct 29 23:30:00.757400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.755439 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftkqt\" (UniqueName: \"kubernetes.io/projected/1c2a4a82-5ab9-4c15-8a1d-54da076ff3db-kube-api-access-ftkqt\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w\" (UID: \"1c2a4a82-5ab9-4c15-8a1d-54da076ff3db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" Oct 29 23:30:00.757400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.755576 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlzhc\" (UniqueName: \"kubernetes.io/projected/cecae6fc-a464-491b-945e-7344fca3eacd-kube-api-access-zlzhc\") pod \"maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns\" (UID: \"cecae6fc-a464-491b-945e-7344fca3eacd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns" Oct 29 23:30:00.757400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.755612 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdv9m\" (UniqueName: \"kubernetes.io/projected/04c2e336-3da0-4816-8313-e8fdeb0edb5f-kube-api-access-wdv9m\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g\" (UID: \"04c2e336-3da0-4816-8313-e8fdeb0edb5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g" Oct 29 23:30:00.757400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.755676 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77571d53-cd5a-4b79-922b-3884bf5f7fcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv\" (UID: \"77571d53-cd5a-4b79-922b-3884bf5f7fcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv" Oct 29 23:30:00.757400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.755743 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f3e7bb-6ca1-46a8-a146-cb7c5690f377-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8\" (UID: \"e2f3e7bb-6ca1-46a8-a146-cb7c5690f377\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8" Oct 29 23:30:00.757400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.755785 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04c2e336-3da0-4816-8313-e8fdeb0edb5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g\" (UID: \"04c2e336-3da0-4816-8313-e8fdeb0edb5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g" Oct 29 23:30:00.757400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.755813 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cecae6fc-a464-491b-945e-7344fca3eacd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns\" (UID: \"cecae6fc-a464-491b-945e-7344fca3eacd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns" Oct 29 23:30:00.757400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.755881 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9lkh\" (UniqueName: \"kubernetes.io/projected/77571d53-cd5a-4b79-922b-3884bf5f7fcb-kube-api-access-x9lkh\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv\" (UID: \"77571d53-cd5a-4b79-922b-3884bf5f7fcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv" Oct 29 23:30:00.757400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.756859 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2a4a82-5ab9-4c15-8a1d-54da076ff3db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w\" (UID: \"1c2a4a82-5ab9-4c15-8a1d-54da076ff3db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" Oct 29 23:30:00.757400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.756977 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f3e7bb-6ca1-46a8-a146-cb7c5690f377-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8\" (UID: \"e2f3e7bb-6ca1-46a8-a146-cb7c5690f377\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8" Oct 29 23:30:00.758013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.757534 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04c2e336-3da0-4816-8313-e8fdeb0edb5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g\" (UID: \"04c2e336-3da0-4816-8313-e8fdeb0edb5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g" Oct 29 23:30:00.800176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.796733 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" Oct 29 23:30:00.803574 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.800594 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftkqt\" (UniqueName: \"kubernetes.io/projected/1c2a4a82-5ab9-4c15-8a1d-54da076ff3db-kube-api-access-ftkqt\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w\" (UID: \"1c2a4a82-5ab9-4c15-8a1d-54da076ff3db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" Oct 29 23:30:00.803574 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.802833 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpdfx\" (UniqueName: \"kubernetes.io/projected/e2f3e7bb-6ca1-46a8-a146-cb7c5690f377-kube-api-access-cpdfx\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8\" (UID: \"e2f3e7bb-6ca1-46a8-a146-cb7c5690f377\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8" Oct 29 23:30:00.813095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.813062 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdv9m\" (UniqueName: \"kubernetes.io/projected/04c2e336-3da0-4816-8313-e8fdeb0edb5f-kube-api-access-wdv9m\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g\" (UID: \"04c2e336-3da0-4816-8313-e8fdeb0edb5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g" Oct 29 23:30:00.828889 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.828859 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8" Oct 29 23:30:00.863353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.859155 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlzhc\" (UniqueName: \"kubernetes.io/projected/cecae6fc-a464-491b-945e-7344fca3eacd-kube-api-access-zlzhc\") pod \"maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns\" (UID: \"cecae6fc-a464-491b-945e-7344fca3eacd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns" Oct 29 23:30:00.863353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.861650 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77571d53-cd5a-4b79-922b-3884bf5f7fcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv\" (UID: \"77571d53-cd5a-4b79-922b-3884bf5f7fcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv" Oct 29 23:30:00.863353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.862395 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cecae6fc-a464-491b-945e-7344fca3eacd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns\" (UID: \"cecae6fc-a464-491b-945e-7344fca3eacd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns" Oct 29 23:30:00.863353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.862503 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9lkh\" (UniqueName: \"kubernetes.io/projected/77571d53-cd5a-4b79-922b-3884bf5f7fcb-kube-api-access-x9lkh\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv\" (UID: \"77571d53-cd5a-4b79-922b-3884bf5f7fcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv" Oct 29 23:30:00.863353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.863061 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77571d53-cd5a-4b79-922b-3884bf5f7fcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv\" (UID: \"77571d53-cd5a-4b79-922b-3884bf5f7fcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv" Oct 29 23:30:00.863731 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.863603 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cecae6fc-a464-491b-945e-7344fca3eacd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns\" (UID: \"cecae6fc-a464-491b-945e-7344fca3eacd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns" Oct 29 23:30:00.864994 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.864902 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95wd9\" (UniqueName: \"kubernetes.io/projected/595f0d6b-00c5-4a95-8881-8191e7de38ff-kube-api-access-95wd9\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh\" (UID: \"595f0d6b-00c5-4a95-8881-8191e7de38ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" Oct 29 23:30:00.887262 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.883830 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" Oct 29 23:30:00.890302 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.890274 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g" Oct 29 23:30:00.895299 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.895272 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9lkh\" (UniqueName: \"kubernetes.io/projected/77571d53-cd5a-4b79-922b-3884bf5f7fcb-kube-api-access-x9lkh\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv\" (UID: \"77571d53-cd5a-4b79-922b-3884bf5f7fcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv" Oct 29 23:30:00.923406 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.923168 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv" Oct 29 23:30:00.946224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:00.946011 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlzhc\" (UniqueName: \"kubernetes.io/projected/cecae6fc-a464-491b-945e-7344fca3eacd-kube-api-access-zlzhc\") pod \"maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns\" (UID: \"cecae6fc-a464-491b-945e-7344fca3eacd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns" Oct 29 23:30:01.134267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:01.124534 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" Oct 29 23:30:01.190205 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:01.185271 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs"] Oct 29 23:30:01.253838 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:01.228703 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns" Oct 29 23:30:01.343064 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:01.342968 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" event={"ID":"a2de74b6-1d73-4ab4-adaa-e081fbc0240d","Type":"ContainerStarted","Data":"88d29f14613ec6d9d003e61f32e588fb0e7fc3d3089bc91ae193acca2209cd46"} Oct 29 23:30:01.425090 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:01.421741 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf"] Oct 29 23:30:01.427182 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:30:01.422249 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2a04ca_f403_49e9_bd0c_f9f9394e188b.slice/crio-6390790e34d44b529a938ff4120f3bc190c9d7dd47dba00a0ca84c71080bc20e WatchSource:0}: Error finding container 6390790e34d44b529a938ff4120f3bc190c9d7dd47dba00a0ca84c71080bc20e: Status 404 returned error can't find the container with id 6390790e34d44b529a938ff4120f3bc190c9d7dd47dba00a0ca84c71080bc20e Oct 29 23:30:01.590616 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:01.590583 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp"] Oct 29 23:30:01.679905 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:01.679853 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6"] Oct 29 23:30:01.689588 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:01.689556 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z"] Oct 29 23:30:01.728526 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:30:01.728481 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13000cb3_436f_4fec_bb70_7e09a449dec0.slice/crio-d6f18ccc8839bb0c02401081a262487f6745c5461762432a05dbd419cc4cd293 WatchSource:0}: Error finding container d6f18ccc8839bb0c02401081a262487f6745c5461762432a05dbd419cc4cd293: Status 404 returned error can't find the container with id d6f18ccc8839bb0c02401081a262487f6745c5461762432a05dbd419cc4cd293 Oct 29 23:30:01.768615 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:01.768528 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd"] Oct 29 23:30:01.775666 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:01.775634 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh"] Oct 29 23:30:01.780350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:01.780321 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w"] Oct 29 23:30:01.812544 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:01.806678 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6"] Oct 29 23:30:01.816412 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:01.816381 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv"] Oct 29 23:30:01.821254 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:01.821223 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns"] Oct 29 23:30:01.826227 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:30:01.826180 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e01bb8e_5fd2_4f7c_acd3_aa87827f3cb8.slice/crio-e6704358d51a52ef8d2fa680e7aba0ba97ffebe70adc7a6f005030921b6e715a WatchSource:0}: Error finding container e6704358d51a52ef8d2fa680e7aba0ba97ffebe70adc7a6f005030921b6e715a: Status 404 returned error can't find the container with id e6704358d51a52ef8d2fa680e7aba0ba97ffebe70adc7a6f005030921b6e715a Oct 29 23:30:01.828001 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:30:01.827976 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea3ec3f_906d_4185_b80b_00c0f50bee8e.slice/crio-540e165c81964e429301998d2b7404786bbd8c411dcee5d9a9e10c319101cf02 WatchSource:0}: Error finding container 540e165c81964e429301998d2b7404786bbd8c411dcee5d9a9e10c319101cf02: Status 404 returned error can't find the container with id 540e165c81964e429301998d2b7404786bbd8c411dcee5d9a9e10c319101cf02 Oct 29 23:30:01.830274 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:30:01.830245 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595f0d6b_00c5_4a95_8881_8191e7de38ff.slice/crio-48a72e7212e707826615698215b657f7795e9fcc06b1798590f2a68692b356ea WatchSource:0}: Error finding container 48a72e7212e707826615698215b657f7795e9fcc06b1798590f2a68692b356ea: Status 404 returned error can't find the container with id 48a72e7212e707826615698215b657f7795e9fcc06b1798590f2a68692b356ea Oct 29 23:30:01.831329 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:30:01.831305 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2a4a82_5ab9_4c15_8a1d_54da076ff3db.slice/crio-4984bf0a9b99ff27f8d64f02e5a555bad8636b6ee09620ae3a30ca581dd64fdf WatchSource:0}: Error finding container 4984bf0a9b99ff27f8d64f02e5a555bad8636b6ee09620ae3a30ca581dd64fdf: Status 404 returned error can't find the container with id 4984bf0a9b99ff27f8d64f02e5a555bad8636b6ee09620ae3a30ca581dd64fdf Oct 29 23:30:01.834860 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:30:01.834837 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb091cb_8d53_4b37_835b_591edcba96f1.slice/crio-fc4b5eee4d3ece156be416854ab798199daacbfd6449ef7afd004b5081df1f5c WatchSource:0}: Error finding container fc4b5eee4d3ece156be416854ab798199daacbfd6449ef7afd004b5081df1f5c: Status 404 returned error can't find the container with id fc4b5eee4d3ece156be416854ab798199daacbfd6449ef7afd004b5081df1f5c Oct 29 23:30:01.907898 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:01.907853 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g"] Oct 29 23:30:01.921819 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:30:01.921779 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c2e336_3da0_4816_8313_e8fdeb0edb5f.slice/crio-0364c21e1eb1c93d613aee99515182ab54da320d8dd103ef0fd9a577060d7658 WatchSource:0}: Error finding container 0364c21e1eb1c93d613aee99515182ab54da320d8dd103ef0fd9a577060d7658: Status 404 returned error can't find the container with id 0364c21e1eb1c93d613aee99515182ab54da320d8dd103ef0fd9a577060d7658 Oct 29 23:30:01.934114 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:01.934084 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8"] Oct 29 23:30:03.079626 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.078234 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv" event={"ID":"77571d53-cd5a-4b79-922b-3884bf5f7fcb","Type":"ContainerStarted","Data":"2510a84a120139616426519d79b24cc3faa935a806cf70d8c5eb2ecb0576f5fb"} Oct 29 23:30:03.083216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.083149 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6" event={"ID":"13000cb3-436f-4fec-bb70-7e09a449dec0","Type":"ContainerStarted","Data":"d6f18ccc8839bb0c02401081a262487f6745c5461762432a05dbd419cc4cd293"} Oct 29 23:30:03.102335 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.102298 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" event={"ID":"8e2a04ca-f403-49e9-bd0c-f9f9394e188b","Type":"ContainerStarted","Data":"891ec3a7fe7f7ed75a2cb38bb3b8486d200308c65cc23cbd36092a92afc308d3"} Oct 29 23:30:03.103091 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.102389 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" event={"ID":"8e2a04ca-f403-49e9-bd0c-f9f9394e188b","Type":"ContainerStarted","Data":"6390790e34d44b529a938ff4120f3bc190c9d7dd47dba00a0ca84c71080bc20e"} Oct 29 23:30:03.120668 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.120628 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8" event={"ID":"e2f3e7bb-6ca1-46a8-a146-cb7c5690f377","Type":"ContainerStarted","Data":"963ea0d28d5d5813ab99279a04bd0c89edd42a5932ce5a20a8c9a4188c4790c0"} Oct 29 23:30:03.150966 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.150922 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" event={"ID":"9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8","Type":"ContainerStarted","Data":"e6704358d51a52ef8d2fa680e7aba0ba97ffebe70adc7a6f005030921b6e715a"} Oct 29 23:30:03.154001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.153962 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns" event={"ID":"cecae6fc-a464-491b-945e-7344fca3eacd","Type":"ContainerStarted","Data":"01485237cca941fcbe3f2c3eaa08506618522659172120ee45f1d0de8ca56285"} Oct 29 23:30:03.157952 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.157918 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" event={"ID":"a2de74b6-1d73-4ab4-adaa-e081fbc0240d","Type":"ContainerStarted","Data":"47ac9ae2c4f59a7e4b5c9cf7230e1df14830d0ce2a8814c2c2128a539d3320a5"} Oct 29 23:30:03.158490 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.158468 5057 scope.go:122] "RemoveContainer" containerID="47ac9ae2c4f59a7e4b5c9cf7230e1df14830d0ce2a8814c2c2128a539d3320a5" Oct 29 23:30:03.170590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.170533 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd" event={"ID":"eea3ec3f-906d-4185-b80b-00c0f50bee8e","Type":"ContainerStarted","Data":"540e165c81964e429301998d2b7404786bbd8c411dcee5d9a9e10c319101cf02"} Oct 29 23:30:03.189142 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.189077 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" event={"ID":"7eb091cb-8d53-4b37-835b-591edcba96f1","Type":"ContainerStarted","Data":"fc4b5eee4d3ece156be416854ab798199daacbfd6449ef7afd004b5081df1f5c"} Oct 29 23:30:03.193938 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.193901 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" event={"ID":"595f0d6b-00c5-4a95-8881-8191e7de38ff","Type":"ContainerStarted","Data":"48a72e7212e707826615698215b657f7795e9fcc06b1798590f2a68692b356ea"} Oct 29 23:30:03.196591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.196562 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" event={"ID":"1c2a4a82-5ab9-4c15-8a1d-54da076ff3db","Type":"ContainerStarted","Data":"4984bf0a9b99ff27f8d64f02e5a555bad8636b6ee09620ae3a30ca581dd64fdf"} Oct 29 23:30:03.198536 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.198478 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" podStartSLOduration=3.198460184 podStartE2EDuration="3.198460184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:03.176842153 +0000 UTC m=+2849.828340684" watchObservedRunningTime="2026-10-29 23:30:03.198460184 +0000 UTC m=+2849.849958714" Oct 29 23:30:03.201290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.201269 5057 scope.go:122] "RemoveContainer" containerID="c1110b73cf711c6f2bdc77dc91fa3355d96bb5463ddb54549e3f18dc9471ae96" Oct 29 23:30:03.201406 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.201263 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" event={"ID":"664c869e-4b17-41bb-946b-d2ead80eff08","Type":"ContainerStarted","Data":"c1110b73cf711c6f2bdc77dc91fa3355d96bb5463ddb54549e3f18dc9471ae96"} Oct 29 23:30:03.201527 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.201403 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" event={"ID":"664c869e-4b17-41bb-946b-d2ead80eff08","Type":"ContainerStarted","Data":"370195e75e9ebcbe8f25989fb6aabcce7d64ddfb48677181d029af955e20f57b"} Oct 29 23:30:03.203435 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.203408 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g" event={"ID":"04c2e336-3da0-4816-8313-e8fdeb0edb5f","Type":"ContainerStarted","Data":"0364c21e1eb1c93d613aee99515182ab54da320d8dd103ef0fd9a577060d7658"} Oct 29 23:30:03.300075 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.299995 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" podStartSLOduration=3.299975032 podStartE2EDuration="3.299975032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:03.298295374 +0000 UTC m=+2849.949793907" watchObservedRunningTime="2026-10-29 23:30:03.299975032 +0000 UTC m=+2849.951473563" Oct 29 23:30:03.346075 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.346000 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" podStartSLOduration=3.345985211 podStartE2EDuration="3.345985211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:03.343590031 +0000 UTC m=+2849.995088561" watchObservedRunningTime="2026-10-29 23:30:03.345985211 +0000 UTC m=+2849.997483741" Oct 29 23:30:03.458870 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:03.456479 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g" podStartSLOduration=3.456457825 podStartE2EDuration="3.456457825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:03.453544921 +0000 UTC m=+2850.105043453" watchObservedRunningTime="2026-10-29 23:30:03.456457825 +0000 UTC m=+2850.107956356" Oct 29 23:30:04.340538 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:04.340494 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:30:04.363649 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.363612 5057 generic.go:356] "Generic (PLEG): container finished" podID="7eb091cb-8d53-4b37-835b-591edcba96f1" containerID="1088a0c8709c414c7e7ced78b52b16edcc02da2d2a3d2f2c44505441c554f8fd" exitCode=60 Oct 29 23:30:04.363850 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.363729 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" event={"ID":"7eb091cb-8d53-4b37-835b-591edcba96f1","Type":"ContainerDied","Data":"1088a0c8709c414c7e7ced78b52b16edcc02da2d2a3d2f2c44505441c554f8fd"} Oct 29 23:30:04.364983 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.364350 5057 scope.go:122] "RemoveContainer" containerID="1088a0c8709c414c7e7ced78b52b16edcc02da2d2a3d2f2c44505441c554f8fd" Oct 29 23:30:04.374034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.371970 5057 generic.go:356] "Generic (PLEG): container finished" podID="595f0d6b-00c5-4a95-8881-8191e7de38ff" containerID="8a0bd5965de804da30f8a5ba72f920d55df1cd64f9471b9006487107065dbfd1" exitCode=60 Oct 29 23:30:04.374034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.372109 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" event={"ID":"595f0d6b-00c5-4a95-8881-8191e7de38ff","Type":"ContainerDied","Data":"8a0bd5965de804da30f8a5ba72f920d55df1cd64f9471b9006487107065dbfd1"} Oct 29 23:30:04.374034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.372609 5057 scope.go:122] "RemoveContainer" containerID="8a0bd5965de804da30f8a5ba72f920d55df1cd64f9471b9006487107065dbfd1" Oct 29 23:30:04.376151 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.376121 5057 generic.go:356] "Generic (PLEG): container finished" podID="1c2a4a82-5ab9-4c15-8a1d-54da076ff3db" containerID="100cc1a32eb630543131bc8d82327b6383d255da04710657f764fe67b91af91c" exitCode=60 Oct 29 23:30:04.378310 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.376264 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" event={"ID":"1c2a4a82-5ab9-4c15-8a1d-54da076ff3db","Type":"ContainerDied","Data":"100cc1a32eb630543131bc8d82327b6383d255da04710657f764fe67b91af91c"} Oct 29 23:30:04.378310 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.376706 5057 scope.go:122] "RemoveContainer" containerID="100cc1a32eb630543131bc8d82327b6383d255da04710657f764fe67b91af91c" Oct 29 23:30:04.379300 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.379274 5057 generic.go:356] "Generic (PLEG): container finished" podID="664c869e-4b17-41bb-946b-d2ead80eff08" containerID="c1110b73cf711c6f2bdc77dc91fa3355d96bb5463ddb54549e3f18dc9471ae96" exitCode=60 Oct 29 23:30:04.379397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.379352 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" event={"ID":"664c869e-4b17-41bb-946b-d2ead80eff08","Type":"ContainerDied","Data":"c1110b73cf711c6f2bdc77dc91fa3355d96bb5463ddb54549e3f18dc9471ae96"} Oct 29 23:30:04.382094 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.382071 5057 generic.go:356] "Generic (PLEG): container finished" podID="04c2e336-3da0-4816-8313-e8fdeb0edb5f" containerID="1bd31266bf9d5058a0b26cd01e5eeed2f34bc646d22007a834c4b6bc9952a06b" exitCode=60 Oct 29 23:30:04.382277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.382178 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g" event={"ID":"04c2e336-3da0-4816-8313-e8fdeb0edb5f","Type":"ContainerDied","Data":"1bd31266bf9d5058a0b26cd01e5eeed2f34bc646d22007a834c4b6bc9952a06b"} Oct 29 23:30:04.382514 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.382495 5057 scope.go:122] "RemoveContainer" containerID="1bd31266bf9d5058a0b26cd01e5eeed2f34bc646d22007a834c4b6bc9952a06b" Oct 29 23:30:04.391508 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.391487 5057 generic.go:356] "Generic (PLEG): container finished" podID="77571d53-cd5a-4b79-922b-3884bf5f7fcb" containerID="dfa97f3bd8d6b89a71815bb4c7fd7996fcde3fce1eb819ea8ab9c014f97db8d7" exitCode=60 Oct 29 23:30:04.391963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.391944 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv" event={"ID":"77571d53-cd5a-4b79-922b-3884bf5f7fcb","Type":"ContainerDied","Data":"dfa97f3bd8d6b89a71815bb4c7fd7996fcde3fce1eb819ea8ab9c014f97db8d7"} Oct 29 23:30:04.392229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.392170 5057 scope.go:122] "RemoveContainer" containerID="dfa97f3bd8d6b89a71815bb4c7fd7996fcde3fce1eb819ea8ab9c014f97db8d7" Oct 29 23:30:04.395020 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.394988 5057 generic.go:356] "Generic (PLEG): container finished" podID="13000cb3-436f-4fec-bb70-7e09a449dec0" containerID="10962ca30caf96adfa5185cafe29271c3522791e7e77114e8e6465f49f810a8b" exitCode=60 Oct 29 23:30:04.395123 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.395094 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6" event={"ID":"13000cb3-436f-4fec-bb70-7e09a449dec0","Type":"ContainerDied","Data":"10962ca30caf96adfa5185cafe29271c3522791e7e77114e8e6465f49f810a8b"} Oct 29 23:30:04.395942 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.395546 5057 scope.go:122] "RemoveContainer" containerID="10962ca30caf96adfa5185cafe29271c3522791e7e77114e8e6465f49f810a8b" Oct 29 23:30:04.397618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.397495 5057 generic.go:356] "Generic (PLEG): container finished" podID="8e2a04ca-f403-49e9-bd0c-f9f9394e188b" containerID="891ec3a7fe7f7ed75a2cb38bb3b8486d200308c65cc23cbd36092a92afc308d3" exitCode=60 Oct 29 23:30:04.397618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.397587 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" event={"ID":"8e2a04ca-f403-49e9-bd0c-f9f9394e188b","Type":"ContainerDied","Data":"891ec3a7fe7f7ed75a2cb38bb3b8486d200308c65cc23cbd36092a92afc308d3"} Oct 29 23:30:04.397983 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.397963 5057 scope.go:122] "RemoveContainer" containerID="891ec3a7fe7f7ed75a2cb38bb3b8486d200308c65cc23cbd36092a92afc308d3" Oct 29 23:30:04.402316 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.402295 5057 generic.go:356] "Generic (PLEG): container finished" podID="e2f3e7bb-6ca1-46a8-a146-cb7c5690f377" containerID="57401704a6f322b0659d773b87647b046553933ce22723da13926d3a636af574" exitCode=60 Oct 29 23:30:04.402507 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.402449 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8" event={"ID":"e2f3e7bb-6ca1-46a8-a146-cb7c5690f377","Type":"ContainerDied","Data":"57401704a6f322b0659d773b87647b046553933ce22723da13926d3a636af574"} Oct 29 23:30:04.405291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.402709 5057 scope.go:122] "RemoveContainer" containerID="57401704a6f322b0659d773b87647b046553933ce22723da13926d3a636af574" Oct 29 23:30:04.407122 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.407096 5057 generic.go:356] "Generic (PLEG): container finished" podID="9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8" containerID="6e6c9e5318c6a7982cf9884ca5cfab2fa941b4a0278864dcc973e790c502300c" exitCode=60 Oct 29 23:30:04.407327 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.407306 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" event={"ID":"9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8","Type":"ContainerDied","Data":"6e6c9e5318c6a7982cf9884ca5cfab2fa941b4a0278864dcc973e790c502300c"} Oct 29 23:30:04.407928 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.407906 5057 scope.go:122] "RemoveContainer" containerID="6e6c9e5318c6a7982cf9884ca5cfab2fa941b4a0278864dcc973e790c502300c" Oct 29 23:30:04.411522 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.411495 5057 generic.go:356] "Generic (PLEG): container finished" podID="cecae6fc-a464-491b-945e-7344fca3eacd" containerID="6b0eb8ae776ff40b96ed0338dc05e4814550eb797cde139724e9606e2f8e2fc9" exitCode=60 Oct 29 23:30:04.411718 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.411699 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns" event={"ID":"cecae6fc-a464-491b-945e-7344fca3eacd","Type":"ContainerDied","Data":"6b0eb8ae776ff40b96ed0338dc05e4814550eb797cde139724e9606e2f8e2fc9"} Oct 29 23:30:04.414293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.414273 5057 scope.go:122] "RemoveContainer" containerID="6b0eb8ae776ff40b96ed0338dc05e4814550eb797cde139724e9606e2f8e2fc9" Oct 29 23:30:04.418547 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.418527 5057 generic.go:356] "Generic (PLEG): container finished" podID="a2de74b6-1d73-4ab4-adaa-e081fbc0240d" containerID="47ac9ae2c4f59a7e4b5c9cf7230e1df14830d0ce2a8814c2c2128a539d3320a5" exitCode=60 Oct 29 23:30:04.418694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.418678 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" event={"ID":"a2de74b6-1d73-4ab4-adaa-e081fbc0240d","Type":"ContainerDied","Data":"47ac9ae2c4f59a7e4b5c9cf7230e1df14830d0ce2a8814c2c2128a539d3320a5"} Oct 29 23:30:04.418740 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.418719 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" event={"ID":"a2de74b6-1d73-4ab4-adaa-e081fbc0240d","Type":"ContainerStarted","Data":"aca325121243eb93c173ddb4ce63b22ae90843646962bc6970cb95fd25c963cf"} Oct 29 23:30:04.420712 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.420686 5057 generic.go:356] "Generic (PLEG): container finished" podID="eea3ec3f-906d-4185-b80b-00c0f50bee8e" containerID="d4261149868432fd6fb9b619b72a9f4969bcf0b95e779bf7777fc4d3acbae3de" exitCode=60 Oct 29 23:30:04.420782 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.420758 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd" event={"ID":"eea3ec3f-906d-4185-b80b-00c0f50bee8e","Type":"ContainerDied","Data":"d4261149868432fd6fb9b619b72a9f4969bcf0b95e779bf7777fc4d3acbae3de"} Oct 29 23:30:04.421101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.421080 5057 scope.go:122] "RemoveContainer" containerID="d4261149868432fd6fb9b619b72a9f4969bcf0b95e779bf7777fc4d3acbae3de" Oct 29 23:30:04.927440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:04.927375 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" podStartSLOduration=4.927353274 podStartE2EDuration="4.927353274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:04.919528962 +0000 UTC m=+2851.571027493" watchObservedRunningTime="2026-10-29 23:30:04.927353274 +0000 UTC m=+2851.578851808" Oct 29 23:30:05.954361 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:05.954329 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" event={"ID":"595f0d6b-00c5-4a95-8881-8191e7de38ff","Type":"ContainerStarted","Data":"9a05eec2ebc30904e1601967da99cbd78c985f58dfb1b28aac64dd640f09c3df"} Oct 29 23:30:06.304875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:06.294141 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" event={"ID":"1c2a4a82-5ab9-4c15-8a1d-54da076ff3db","Type":"ContainerStarted","Data":"36ff7744f46c0a90d5a18dd1a977856f8292d73af475191141d6225492c53782"} Oct 29 23:30:06.387049 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:06.387007 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" event={"ID":"664c869e-4b17-41bb-946b-d2ead80eff08","Type":"ContainerStarted","Data":"28d07885a76f1ce9bfe946a83e8d8dcf60d7fb8b868ad42f573fec693321a603"} Oct 29 23:30:06.560652 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:06.541732 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g" event={"ID":"04c2e336-3da0-4816-8313-e8fdeb0edb5f","Type":"ContainerStarted","Data":"7e959b25bce9b4dba9b1f8768e66e62f0e642b8bda63db23d5dc45855e691ebc"} Oct 29 23:30:06.604210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:06.603564 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" podStartSLOduration=6.603544016 podStartE2EDuration="6.603544016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:06.466924849 +0000 UTC m=+2853.118423404" watchObservedRunningTime="2026-10-29 23:30:06.603544016 +0000 UTC m=+2853.255042548" Oct 29 23:30:06.639017 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:06.638977 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" event={"ID":"8e2a04ca-f403-49e9-bd0c-f9f9394e188b","Type":"ContainerStarted","Data":"c378f14c204c5e1735ae82f983c08a752f6cff8eb44d712837e68d6d22167b5a"} Oct 29 23:30:06.733111 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:06.733069 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" event={"ID":"9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8","Type":"ContainerStarted","Data":"29e949ca7ea7435b6b6e6c30bf1f06c2483e6d533178cb4c0919ced9b7814664"} Oct 29 23:30:06.734148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:06.734101 5057 scope.go:122] "RemoveContainer" containerID="29e949ca7ea7435b6b6e6c30bf1f06c2483e6d533178cb4c0919ced9b7814664" Oct 29 23:30:06.735155 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:06.735125 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z_odh-ai-gateway-infra(9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" podUID="9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8" Oct 29 23:30:06.773731 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:06.773685 5057 generic.go:356] "Generic (PLEG): container finished" podID="a2de74b6-1d73-4ab4-adaa-e081fbc0240d" containerID="aca325121243eb93c173ddb4ce63b22ae90843646962bc6970cb95fd25c963cf" exitCode=60 Oct 29 23:30:06.773970 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:06.773944 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" event={"ID":"a2de74b6-1d73-4ab4-adaa-e081fbc0240d","Type":"ContainerDied","Data":"aca325121243eb93c173ddb4ce63b22ae90843646962bc6970cb95fd25c963cf"} Oct 29 23:30:06.774050 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:06.774025 5057 scope.go:122] "RemoveContainer" containerID="47ac9ae2c4f59a7e4b5c9cf7230e1df14830d0ce2a8814c2c2128a539d3320a5" Oct 29 23:30:06.774805 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:06.774781 5057 scope.go:122] "RemoveContainer" containerID="aca325121243eb93c173ddb4ce63b22ae90843646962bc6970cb95fd25c963cf" Oct 29 23:30:06.775857 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:06.775583 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs_odh-ai-gateway-infra(a2de74b6-1d73-4ab4-adaa-e081fbc0240d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" podUID="a2de74b6-1d73-4ab4-adaa-e081fbc0240d" Oct 29 23:30:06.778676 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:06.778608 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" event={"ID":"7eb091cb-8d53-4b37-835b-591edcba96f1","Type":"ContainerStarted","Data":"99ed2333cace3765aaf5c55d499e51b3b4cb6900dc95bf0c480e425542837dea"} Oct 29 23:30:06.780060 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:06.779569 5057 scope.go:122] "RemoveContainer" containerID="99ed2333cace3765aaf5c55d499e51b3b4cb6900dc95bf0c480e425542837dea" Oct 29 23:30:06.780557 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:06.780527 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6_odh-ai-gateway-infra(7eb091cb-8d53-4b37-835b-591edcba96f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" podUID="7eb091cb-8d53-4b37-835b-591edcba96f1" Oct 29 23:30:06.794999 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:06.794800 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8" podStartSLOduration=6.794785462 podStartE2EDuration="6.794785462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:06.763311098 +0000 UTC m=+2853.414809630" watchObservedRunningTime="2026-10-29 23:30:06.794785462 +0000 UTC m=+2853.446284037" Oct 29 23:30:07.793366 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.793330 5057 generic.go:356] "Generic (PLEG): container finished" podID="13000cb3-436f-4fec-bb70-7e09a449dec0" containerID="b7a221f714d8db67ee8565967c2658d1df4cd3a00bd533fd7c6efadd9e2f5766" exitCode=60 Oct 29 23:30:07.793864 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.793451 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6" event={"ID":"13000cb3-436f-4fec-bb70-7e09a449dec0","Type":"ContainerDied","Data":"b7a221f714d8db67ee8565967c2658d1df4cd3a00bd533fd7c6efadd9e2f5766"} Oct 29 23:30:07.793864 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.793512 5057 scope.go:122] "RemoveContainer" containerID="10962ca30caf96adfa5185cafe29271c3522791e7e77114e8e6465f49f810a8b" Oct 29 23:30:07.793864 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.793840 5057 scope.go:122] "RemoveContainer" containerID="b7a221f714d8db67ee8565967c2658d1df4cd3a00bd533fd7c6efadd9e2f5766" Oct 29 23:30:07.794414 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:07.794379 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6_odh-ai-gateway-infra(13000cb3-436f-4fec-bb70-7e09a449dec0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6" podUID="13000cb3-436f-4fec-bb70-7e09a449dec0" Oct 29 23:30:07.796619 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.796592 5057 generic.go:356] "Generic (PLEG): container finished" podID="8e2a04ca-f403-49e9-bd0c-f9f9394e188b" containerID="c378f14c204c5e1735ae82f983c08a752f6cff8eb44d712837e68d6d22167b5a" exitCode=60 Oct 29 23:30:07.796726 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.796670 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" event={"ID":"8e2a04ca-f403-49e9-bd0c-f9f9394e188b","Type":"ContainerDied","Data":"c378f14c204c5e1735ae82f983c08a752f6cff8eb44d712837e68d6d22167b5a"} Oct 29 23:30:07.797017 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.796993 5057 scope.go:122] "RemoveContainer" containerID="c378f14c204c5e1735ae82f983c08a752f6cff8eb44d712837e68d6d22167b5a" Oct 29 23:30:07.797502 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:07.797478 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf_odh-ai-gateway-infra(8e2a04ca-f403-49e9-bd0c-f9f9394e188b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" podUID="8e2a04ca-f403-49e9-bd0c-f9f9394e188b" Oct 29 23:30:07.800651 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.800620 5057 generic.go:356] "Generic (PLEG): container finished" podID="e2f3e7bb-6ca1-46a8-a146-cb7c5690f377" containerID="c08ffcca76645ba20f222b0d9d7e5b663f9aae99e67958a89cdb1307a985ba3d" exitCode=60 Oct 29 23:30:07.800768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.800693 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8" event={"ID":"e2f3e7bb-6ca1-46a8-a146-cb7c5690f377","Type":"ContainerDied","Data":"c08ffcca76645ba20f222b0d9d7e5b663f9aae99e67958a89cdb1307a985ba3d"} Oct 29 23:30:07.801230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.801154 5057 scope.go:122] "RemoveContainer" containerID="c08ffcca76645ba20f222b0d9d7e5b663f9aae99e67958a89cdb1307a985ba3d" Oct 29 23:30:07.801898 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:07.801862 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8_odh-ai-gateway-infra(e2f3e7bb-6ca1-46a8-a146-cb7c5690f377)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8" podUID="e2f3e7bb-6ca1-46a8-a146-cb7c5690f377" Oct 29 23:30:07.802977 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.802954 5057 generic.go:356] "Generic (PLEG): container finished" podID="9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8" containerID="29e949ca7ea7435b6b6e6c30bf1f06c2483e6d533178cb4c0919ced9b7814664" exitCode=60 Oct 29 23:30:07.803065 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.803036 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" event={"ID":"9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8","Type":"ContainerDied","Data":"29e949ca7ea7435b6b6e6c30bf1f06c2483e6d533178cb4c0919ced9b7814664"} Oct 29 23:30:07.803435 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.803414 5057 scope.go:122] "RemoveContainer" containerID="29e949ca7ea7435b6b6e6c30bf1f06c2483e6d533178cb4c0919ced9b7814664" Oct 29 23:30:07.803806 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:07.803783 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z_odh-ai-gateway-infra(9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" podUID="9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8" Oct 29 23:30:07.805772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.805751 5057 generic.go:356] "Generic (PLEG): container finished" podID="cecae6fc-a464-491b-945e-7344fca3eacd" containerID="cc41d0ce57cace3d0c30b3f6ebb5e57b32dcc15124628f16eeb426aa6481d9ea" exitCode=60 Oct 29 23:30:07.805891 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.805834 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns" event={"ID":"cecae6fc-a464-491b-945e-7344fca3eacd","Type":"ContainerDied","Data":"cc41d0ce57cace3d0c30b3f6ebb5e57b32dcc15124628f16eeb426aa6481d9ea"} Oct 29 23:30:07.806169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.806145 5057 scope.go:122] "RemoveContainer" containerID="cc41d0ce57cace3d0c30b3f6ebb5e57b32dcc15124628f16eeb426aa6481d9ea" Oct 29 23:30:07.806951 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:07.806561 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns_odh-ai-gateway-infra(cecae6fc-a464-491b-945e-7344fca3eacd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns" podUID="cecae6fc-a464-491b-945e-7344fca3eacd" Oct 29 23:30:07.808289 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.808268 5057 scope.go:122] "RemoveContainer" containerID="aca325121243eb93c173ddb4ce63b22ae90843646962bc6970cb95fd25c963cf" Oct 29 23:30:07.808611 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:07.808585 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs_odh-ai-gateway-infra(a2de74b6-1d73-4ab4-adaa-e081fbc0240d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" podUID="a2de74b6-1d73-4ab4-adaa-e081fbc0240d" Oct 29 23:30:07.809981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.809960 5057 generic.go:356] "Generic (PLEG): container finished" podID="eea3ec3f-906d-4185-b80b-00c0f50bee8e" containerID="12369f5bb231ea4710d80139932fa7beb60b0afd6f276fbfe48e64b3c0cde59a" exitCode=60 Oct 29 23:30:07.810064 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.810036 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd" event={"ID":"eea3ec3f-906d-4185-b80b-00c0f50bee8e","Type":"ContainerDied","Data":"12369f5bb231ea4710d80139932fa7beb60b0afd6f276fbfe48e64b3c0cde59a"} Oct 29 23:30:07.810332 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.810316 5057 scope.go:122] "RemoveContainer" containerID="12369f5bb231ea4710d80139932fa7beb60b0afd6f276fbfe48e64b3c0cde59a" Oct 29 23:30:07.810617 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:07.810593 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd_odh-ai-gateway-infra(eea3ec3f-906d-4185-b80b-00c0f50bee8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd" podUID="eea3ec3f-906d-4185-b80b-00c0f50bee8e" Oct 29 23:30:07.812124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.812100 5057 generic.go:356] "Generic (PLEG): container finished" podID="7eb091cb-8d53-4b37-835b-591edcba96f1" containerID="99ed2333cace3765aaf5c55d499e51b3b4cb6900dc95bf0c480e425542837dea" exitCode=60 Oct 29 23:30:07.812229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.812179 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" event={"ID":"7eb091cb-8d53-4b37-835b-591edcba96f1","Type":"ContainerDied","Data":"99ed2333cace3765aaf5c55d499e51b3b4cb6900dc95bf0c480e425542837dea"} Oct 29 23:30:07.812475 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.812458 5057 scope.go:122] "RemoveContainer" containerID="99ed2333cace3765aaf5c55d499e51b3b4cb6900dc95bf0c480e425542837dea" Oct 29 23:30:07.812792 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:07.812762 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6_odh-ai-gateway-infra(7eb091cb-8d53-4b37-835b-591edcba96f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" podUID="7eb091cb-8d53-4b37-835b-591edcba96f1" Oct 29 23:30:07.813145 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.813127 5057 scope.go:122] "RemoveContainer" containerID="891ec3a7fe7f7ed75a2cb38bb3b8486d200308c65cc23cbd36092a92afc308d3" Oct 29 23:30:07.815178 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.815155 5057 generic.go:356] "Generic (PLEG): container finished" podID="595f0d6b-00c5-4a95-8881-8191e7de38ff" containerID="9a05eec2ebc30904e1601967da99cbd78c985f58dfb1b28aac64dd640f09c3df" exitCode=60 Oct 29 23:30:07.815267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.815181 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" event={"ID":"595f0d6b-00c5-4a95-8881-8191e7de38ff","Type":"ContainerDied","Data":"9a05eec2ebc30904e1601967da99cbd78c985f58dfb1b28aac64dd640f09c3df"} Oct 29 23:30:07.815516 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.815498 5057 scope.go:122] "RemoveContainer" containerID="9a05eec2ebc30904e1601967da99cbd78c985f58dfb1b28aac64dd640f09c3df" Oct 29 23:30:07.815811 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:07.815791 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh_odh-ai-gateway-infra(595f0d6b-00c5-4a95-8881-8191e7de38ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" podUID="595f0d6b-00c5-4a95-8881-8191e7de38ff" Oct 29 23:30:07.817256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.817234 5057 generic.go:356] "Generic (PLEG): container finished" podID="1c2a4a82-5ab9-4c15-8a1d-54da076ff3db" containerID="36ff7744f46c0a90d5a18dd1a977856f8292d73af475191141d6225492c53782" exitCode=60 Oct 29 23:30:07.817337 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.817303 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" event={"ID":"1c2a4a82-5ab9-4c15-8a1d-54da076ff3db","Type":"ContainerDied","Data":"36ff7744f46c0a90d5a18dd1a977856f8292d73af475191141d6225492c53782"} Oct 29 23:30:07.817496 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.817483 5057 scope.go:122] "RemoveContainer" containerID="36ff7744f46c0a90d5a18dd1a977856f8292d73af475191141d6225492c53782" Oct 29 23:30:07.817728 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:07.817707 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w_odh-ai-gateway-infra(1c2a4a82-5ab9-4c15-8a1d-54da076ff3db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" podUID="1c2a4a82-5ab9-4c15-8a1d-54da076ff3db" Oct 29 23:30:07.819230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.819206 5057 generic.go:356] "Generic (PLEG): container finished" podID="664c869e-4b17-41bb-946b-d2ead80eff08" containerID="28d07885a76f1ce9bfe946a83e8d8dcf60d7fb8b868ad42f573fec693321a603" exitCode=60 Oct 29 23:30:07.819343 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.819293 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" event={"ID":"664c869e-4b17-41bb-946b-d2ead80eff08","Type":"ContainerDied","Data":"28d07885a76f1ce9bfe946a83e8d8dcf60d7fb8b868ad42f573fec693321a603"} Oct 29 23:30:07.819562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.819543 5057 scope.go:122] "RemoveContainer" containerID="28d07885a76f1ce9bfe946a83e8d8dcf60d7fb8b868ad42f573fec693321a603" Oct 29 23:30:07.819793 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:07.819771 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp_odh-ai-gateway-infra(664c869e-4b17-41bb-946b-d2ead80eff08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" podUID="664c869e-4b17-41bb-946b-d2ead80eff08" Oct 29 23:30:07.821141 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.821123 5057 generic.go:356] "Generic (PLEG): container finished" podID="04c2e336-3da0-4816-8313-e8fdeb0edb5f" containerID="7e959b25bce9b4dba9b1f8768e66e62f0e642b8bda63db23d5dc45855e691ebc" exitCode=60 Oct 29 23:30:07.821302 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.821210 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g" event={"ID":"04c2e336-3da0-4816-8313-e8fdeb0edb5f","Type":"ContainerDied","Data":"7e959b25bce9b4dba9b1f8768e66e62f0e642b8bda63db23d5dc45855e691ebc"} Oct 29 23:30:07.821406 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.821390 5057 scope.go:122] "RemoveContainer" containerID="7e959b25bce9b4dba9b1f8768e66e62f0e642b8bda63db23d5dc45855e691ebc" Oct 29 23:30:07.821627 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:07.821608 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g_odh-ai-gateway-infra(04c2e336-3da0-4816-8313-e8fdeb0edb5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g" podUID="04c2e336-3da0-4816-8313-e8fdeb0edb5f" Oct 29 23:30:07.823271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.823251 5057 generic.go:356] "Generic (PLEG): container finished" podID="77571d53-cd5a-4b79-922b-3884bf5f7fcb" containerID="c928a17b85d6c7f973512f4896d844c1df3e3981f355b445b60be9afef1fa81a" exitCode=60 Oct 29 23:30:07.823369 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.823306 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv" event={"ID":"77571d53-cd5a-4b79-922b-3884bf5f7fcb","Type":"ContainerDied","Data":"c928a17b85d6c7f973512f4896d844c1df3e3981f355b445b60be9afef1fa81a"} Oct 29 23:30:07.823535 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.823522 5057 scope.go:122] "RemoveContainer" containerID="c928a17b85d6c7f973512f4896d844c1df3e3981f355b445b60be9afef1fa81a" Oct 29 23:30:07.823746 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:07.823731 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv_odh-ai-gateway-infra(77571d53-cd5a-4b79-922b-3884bf5f7fcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv" podUID="77571d53-cd5a-4b79-922b-3884bf5f7fcb" Oct 29 23:30:07.829130 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.829108 5057 scope.go:122] "RemoveContainer" containerID="57401704a6f322b0659d773b87647b046553933ce22723da13926d3a636af574" Oct 29 23:30:07.845475 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.845451 5057 scope.go:122] "RemoveContainer" containerID="6e6c9e5318c6a7982cf9884ca5cfab2fa941b4a0278864dcc973e790c502300c" Oct 29 23:30:07.862661 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.862504 5057 scope.go:122] "RemoveContainer" containerID="6b0eb8ae776ff40b96ed0338dc05e4814550eb797cde139724e9606e2f8e2fc9" Oct 29 23:30:07.880228 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.880201 5057 scope.go:122] "RemoveContainer" containerID="d4261149868432fd6fb9b619b72a9f4969bcf0b95e779bf7777fc4d3acbae3de" Oct 29 23:30:07.896539 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.896519 5057 scope.go:122] "RemoveContainer" containerID="1088a0c8709c414c7e7ced78b52b16edcc02da2d2a3d2f2c44505441c554f8fd" Oct 29 23:30:07.916095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.916063 5057 scope.go:122] "RemoveContainer" containerID="8a0bd5965de804da30f8a5ba72f920d55df1cd64f9471b9006487107065dbfd1" Oct 29 23:30:07.932353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.932318 5057 scope.go:122] "RemoveContainer" containerID="100cc1a32eb630543131bc8d82327b6383d255da04710657f764fe67b91af91c" Oct 29 23:30:07.947948 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.947919 5057 scope.go:122] "RemoveContainer" containerID="c1110b73cf711c6f2bdc77dc91fa3355d96bb5463ddb54549e3f18dc9471ae96" Oct 29 23:30:07.963089 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.963066 5057 scope.go:122] "RemoveContainer" containerID="1bd31266bf9d5058a0b26cd01e5eeed2f34bc646d22007a834c4b6bc9952a06b" Oct 29 23:30:07.978775 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:07.978744 5057 scope.go:122] "RemoveContainer" containerID="dfa97f3bd8d6b89a71815bb4c7fd7996fcde3fce1eb819ea8ab9c014f97db8d7" Oct 29 23:30:08.895076 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:08.895048 5057 scope.go:122] "RemoveContainer" containerID="12369f5bb231ea4710d80139932fa7beb60b0afd6f276fbfe48e64b3c0cde59a" Oct 29 23:30:08.896320 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:08.896290 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd_odh-ai-gateway-infra(eea3ec3f-906d-4185-b80b-00c0f50bee8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd" podUID="eea3ec3f-906d-4185-b80b-00c0f50bee8e" Oct 29 23:30:08.901015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:08.900992 5057 scope.go:122] "RemoveContainer" containerID="99ed2333cace3765aaf5c55d499e51b3b4cb6900dc95bf0c480e425542837dea" Oct 29 23:30:08.901692 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:08.901659 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6_odh-ai-gateway-infra(7eb091cb-8d53-4b37-835b-591edcba96f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" podUID="7eb091cb-8d53-4b37-835b-591edcba96f1" Oct 29 23:30:08.905678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:08.905652 5057 scope.go:122] "RemoveContainer" containerID="9a05eec2ebc30904e1601967da99cbd78c985f58dfb1b28aac64dd640f09c3df" Oct 29 23:30:08.906113 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:08.906087 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh_odh-ai-gateway-infra(595f0d6b-00c5-4a95-8881-8191e7de38ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" podUID="595f0d6b-00c5-4a95-8881-8191e7de38ff" Oct 29 23:30:08.908586 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:08.908566 5057 scope.go:122] "RemoveContainer" containerID="36ff7744f46c0a90d5a18dd1a977856f8292d73af475191141d6225492c53782" Oct 29 23:30:08.909158 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:08.909133 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w_odh-ai-gateway-infra(1c2a4a82-5ab9-4c15-8a1d-54da076ff3db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" podUID="1c2a4a82-5ab9-4c15-8a1d-54da076ff3db" Oct 29 23:30:08.911024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:08.911004 5057 scope.go:122] "RemoveContainer" containerID="28d07885a76f1ce9bfe946a83e8d8dcf60d7fb8b868ad42f573fec693321a603" Oct 29 23:30:08.911477 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:08.911454 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp_odh-ai-gateway-infra(664c869e-4b17-41bb-946b-d2ead80eff08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" podUID="664c869e-4b17-41bb-946b-d2ead80eff08" Oct 29 23:30:08.913970 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:08.913952 5057 scope.go:122] "RemoveContainer" containerID="7e959b25bce9b4dba9b1f8768e66e62f0e642b8bda63db23d5dc45855e691ebc" Oct 29 23:30:08.914513 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:08.914488 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g_odh-ai-gateway-infra(04c2e336-3da0-4816-8313-e8fdeb0edb5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g" podUID="04c2e336-3da0-4816-8313-e8fdeb0edb5f" Oct 29 23:30:08.915882 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:08.915862 5057 scope.go:122] "RemoveContainer" containerID="c928a17b85d6c7f973512f4896d844c1df3e3981f355b445b60be9afef1fa81a" Oct 29 23:30:08.916163 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:08.916140 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv_odh-ai-gateway-infra(77571d53-cd5a-4b79-922b-3884bf5f7fcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv" podUID="77571d53-cd5a-4b79-922b-3884bf5f7fcb" Oct 29 23:30:08.917709 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:08.917689 5057 scope.go:122] "RemoveContainer" containerID="b7a221f714d8db67ee8565967c2658d1df4cd3a00bd533fd7c6efadd9e2f5766" Oct 29 23:30:08.918022 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:08.918001 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6_odh-ai-gateway-infra(13000cb3-436f-4fec-bb70-7e09a449dec0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6" podUID="13000cb3-436f-4fec-bb70-7e09a449dec0" Oct 29 23:30:08.934840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:08.934812 5057 scope.go:122] "RemoveContainer" containerID="c378f14c204c5e1735ae82f983c08a752f6cff8eb44d712837e68d6d22167b5a" Oct 29 23:30:08.935620 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:08.935592 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf_odh-ai-gateway-infra(8e2a04ca-f403-49e9-bd0c-f9f9394e188b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" podUID="8e2a04ca-f403-49e9-bd0c-f9f9394e188b" Oct 29 23:30:08.941468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:08.941448 5057 scope.go:122] "RemoveContainer" containerID="29e949ca7ea7435b6b6e6c30bf1f06c2483e6d533178cb4c0919ced9b7814664" Oct 29 23:30:08.941917 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:08.941896 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z_odh-ai-gateway-infra(9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" podUID="9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8" Oct 29 23:30:08.944077 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:08.944060 5057 scope.go:122] "RemoveContainer" containerID="cc41d0ce57cace3d0c30b3f6ebb5e57b32dcc15124628f16eeb426aa6481d9ea" Oct 29 23:30:08.944499 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:08.944475 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns_odh-ai-gateway-infra(cecae6fc-a464-491b-945e-7344fca3eacd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns" podUID="cecae6fc-a464-491b-945e-7344fca3eacd" Oct 29 23:30:16.500211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:16.476864 5057 scope.go:122] "RemoveContainer" containerID="c378f14c204c5e1735ae82f983c08a752f6cff8eb44d712837e68d6d22167b5a" Oct 29 23:30:16.501389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:16.501361 5057 scope.go:122] "RemoveContainer" containerID="7e959b25bce9b4dba9b1f8768e66e62f0e642b8bda63db23d5dc45855e691ebc" Oct 29 23:30:17.131958 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:17.131929 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" event={"ID":"8e2a04ca-f403-49e9-bd0c-f9f9394e188b","Type":"ContainerStarted","Data":"18c902eafa2b5a72728196ddd7d9480448cc6cffb60617a82e1ade24bdf3f5b5"} Oct 29 23:30:17.473701 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:17.473659 5057 scope.go:122] "RemoveContainer" containerID="29e949ca7ea7435b6b6e6c30bf1f06c2483e6d533178cb4c0919ced9b7814664" Oct 29 23:30:17.474010 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:17.473988 5057 scope.go:122] "RemoveContainer" containerID="36ff7744f46c0a90d5a18dd1a977856f8292d73af475191141d6225492c53782" Oct 29 23:30:18.151027 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:18.150998 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" event={"ID":"1c2a4a82-5ab9-4c15-8a1d-54da076ff3db","Type":"ContainerStarted","Data":"c576f905edb9017c8c463b76dad68e799fd283bb37f662a761b54dd22fb65924"} Oct 29 23:30:18.153427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:18.153399 5057 generic.go:356] "Generic (PLEG): container finished" podID="04c2e336-3da0-4816-8313-e8fdeb0edb5f" containerID="ed3f6319454b549319e0da0c79b4adec23048ec7e39839d96e1c9892b3d62b4e" exitCode=60 Oct 29 23:30:18.153549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:18.153475 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g" event={"ID":"04c2e336-3da0-4816-8313-e8fdeb0edb5f","Type":"ContainerDied","Data":"ed3f6319454b549319e0da0c79b4adec23048ec7e39839d96e1c9892b3d62b4e"} Oct 29 23:30:18.153549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:18.153539 5057 scope.go:122] "RemoveContainer" containerID="7e959b25bce9b4dba9b1f8768e66e62f0e642b8bda63db23d5dc45855e691ebc" Oct 29 23:30:18.153980 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:18.153961 5057 scope.go:122] "RemoveContainer" containerID="ed3f6319454b549319e0da0c79b4adec23048ec7e39839d96e1c9892b3d62b4e" Oct 29 23:30:18.154406 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:18.154377 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g_odh-ai-gateway-infra(04c2e336-3da0-4816-8313-e8fdeb0edb5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g" podUID="04c2e336-3da0-4816-8313-e8fdeb0edb5f" Oct 29 23:30:18.155952 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:18.155930 5057 generic.go:356] "Generic (PLEG): container finished" podID="8e2a04ca-f403-49e9-bd0c-f9f9394e188b" containerID="18c902eafa2b5a72728196ddd7d9480448cc6cffb60617a82e1ade24bdf3f5b5" exitCode=60 Oct 29 23:30:18.156029 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:18.156008 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" event={"ID":"8e2a04ca-f403-49e9-bd0c-f9f9394e188b","Type":"ContainerDied","Data":"18c902eafa2b5a72728196ddd7d9480448cc6cffb60617a82e1ade24bdf3f5b5"} Oct 29 23:30:18.156431 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:18.156410 5057 scope.go:122] "RemoveContainer" containerID="18c902eafa2b5a72728196ddd7d9480448cc6cffb60617a82e1ade24bdf3f5b5" Oct 29 23:30:18.156794 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:18.156763 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf_odh-ai-gateway-infra(8e2a04ca-f403-49e9-bd0c-f9f9394e188b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" podUID="8e2a04ca-f403-49e9-bd0c-f9f9394e188b" Oct 29 23:30:18.158138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:18.158117 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" event={"ID":"9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8","Type":"ContainerStarted","Data":"1d4ee90d88bf7f72a2cedb8969db35e175fbe13ac39dfa635f110a0aec6a7903"} Oct 29 23:30:18.169746 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:18.169717 5057 scope.go:122] "RemoveContainer" containerID="c378f14c204c5e1735ae82f983c08a752f6cff8eb44d712837e68d6d22167b5a" Oct 29 23:30:18.206861 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:18.206834 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf"] Oct 29 23:30:18.284237 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:18.283740 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" podStartSLOduration=18.283723774 podStartE2EDuration="18.283723774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:18.283446619 +0000 UTC m=+2864.934945152" watchObservedRunningTime="2026-10-29 23:30:18.283723774 +0000 UTC m=+2864.935222305" Oct 29 23:30:18.473989 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:18.473953 5057 scope.go:122] "RemoveContainer" containerID="aca325121243eb93c173ddb4ce63b22ae90843646962bc6970cb95fd25c963cf" Oct 29 23:30:18.474327 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:18.474305 5057 scope.go:122] "RemoveContainer" containerID="9a05eec2ebc30904e1601967da99cbd78c985f58dfb1b28aac64dd640f09c3df" Oct 29 23:30:18.474625 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:18.474599 5057 scope.go:122] "RemoveContainer" containerID="28d07885a76f1ce9bfe946a83e8d8dcf60d7fb8b868ad42f573fec693321a603" Oct 29 23:30:18.474716 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:18.474647 5057 scope.go:122] "RemoveContainer" containerID="12369f5bb231ea4710d80139932fa7beb60b0afd6f276fbfe48e64b3c0cde59a" Oct 29 23:30:19.170944 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.170917 5057 generic.go:356] "Generic (PLEG): container finished" podID="9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8" containerID="1d4ee90d88bf7f72a2cedb8969db35e175fbe13ac39dfa635f110a0aec6a7903" exitCode=60 Oct 29 23:30:19.171357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.170985 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" event={"ID":"9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8","Type":"ContainerDied","Data":"1d4ee90d88bf7f72a2cedb8969db35e175fbe13ac39dfa635f110a0aec6a7903"} Oct 29 23:30:19.171357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.171053 5057 scope.go:122] "RemoveContainer" containerID="29e949ca7ea7435b6b6e6c30bf1f06c2483e6d533178cb4c0919ced9b7814664" Oct 29 23:30:19.171532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.171516 5057 scope.go:122] "RemoveContainer" containerID="1d4ee90d88bf7f72a2cedb8969db35e175fbe13ac39dfa635f110a0aec6a7903" Oct 29 23:30:19.171896 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:19.171878 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z_odh-ai-gateway-infra(9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" podUID="9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8" Oct 29 23:30:19.174141 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.174118 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" event={"ID":"a2de74b6-1d73-4ab4-adaa-e081fbc0240d","Type":"ContainerStarted","Data":"2dadea59d9b84f953c184dbd3269857d753ee365301f659e3e74e75b1f23c7bd"} Oct 29 23:30:19.176244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.176215 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd" event={"ID":"eea3ec3f-906d-4185-b80b-00c0f50bee8e","Type":"ContainerStarted","Data":"57804e426c88ddaa11e98272141dfea13a63b477dac87dd5313cc68ca88c7558"} Oct 29 23:30:19.178690 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.178666 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" event={"ID":"595f0d6b-00c5-4a95-8881-8191e7de38ff","Type":"ContainerStarted","Data":"9346031c15b404ed0266dc2aeed2ca00d94f9fecbfa80e1358dddf273fb39d22"} Oct 29 23:30:19.181110 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.181085 5057 generic.go:356] "Generic (PLEG): container finished" podID="1c2a4a82-5ab9-4c15-8a1d-54da076ff3db" containerID="c576f905edb9017c8c463b76dad68e799fd283bb37f662a761b54dd22fb65924" exitCode=60 Oct 29 23:30:19.181245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.181151 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" event={"ID":"1c2a4a82-5ab9-4c15-8a1d-54da076ff3db","Type":"ContainerDied","Data":"c576f905edb9017c8c463b76dad68e799fd283bb37f662a761b54dd22fb65924"} Oct 29 23:30:19.181489 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.181474 5057 scope.go:122] "RemoveContainer" containerID="c576f905edb9017c8c463b76dad68e799fd283bb37f662a761b54dd22fb65924" Oct 29 23:30:19.181787 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:19.181763 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w_odh-ai-gateway-infra(1c2a4a82-5ab9-4c15-8a1d-54da076ff3db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" podUID="1c2a4a82-5ab9-4c15-8a1d-54da076ff3db" Oct 29 23:30:19.183645 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.183622 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" event={"ID":"664c869e-4b17-41bb-946b-d2ead80eff08","Type":"ContainerStarted","Data":"2d0e94fb5de1ec846ba4f5a26f36c33149f7dddd6c9882196690889048f7c02a"} Oct 29 23:30:19.189201 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.189169 5057 scope.go:122] "RemoveContainer" containerID="36ff7744f46c0a90d5a18dd1a977856f8292d73af475191141d6225492c53782" Oct 29 23:30:19.212809 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.212677 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w"] Oct 29 23:30:19.314971 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.314864 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z"] Oct 29 23:30:19.391585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.391557 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g"] Oct 29 23:30:19.475935 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.475486 5057 scope.go:122] "RemoveContainer" containerID="99ed2333cace3765aaf5c55d499e51b3b4cb6900dc95bf0c480e425542837dea" Oct 29 23:30:19.477891 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.476358 5057 scope.go:122] "RemoveContainer" containerID="cc41d0ce57cace3d0c30b3f6ebb5e57b32dcc15124628f16eeb426aa6481d9ea" Oct 29 23:30:19.477891 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.476567 5057 scope.go:122] "RemoveContainer" containerID="c08ffcca76645ba20f222b0d9d7e5b663f9aae99e67958a89cdb1307a985ba3d" Oct 29 23:30:19.477891 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.477241 5057 scope.go:122] "RemoveContainer" containerID="b7a221f714d8db67ee8565967c2658d1df4cd3a00bd533fd7c6efadd9e2f5766" Oct 29 23:30:19.526292 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.525783 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd" podStartSLOduration=19.525758427 podStartE2EDuration="19.525758427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:19.525266443 +0000 UTC m=+2866.176764973" watchObservedRunningTime="2026-10-29 23:30:19.525758427 +0000 UTC m=+2866.177256959" Oct 29 23:30:19.565523 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.565457 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" Oct 29 23:30:19.663967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.663923 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2a04ca-f403-49e9-bd0c-f9f9394e188b-kube-api-access-z54mj\" (UniqueName: \"kubernetes.io/projected/8e2a04ca-f403-49e9-bd0c-f9f9394e188b-kube-api-access-z54mj\") pod \"8e2a04ca-f403-49e9-bd0c-f9f9394e188b\" (UID: \"8e2a04ca-f403-49e9-bd0c-f9f9394e188b\") " Oct 29 23:30:19.664179 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.663994 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e2a04ca-f403-49e9-bd0c-f9f9394e188b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2a04ca-f403-49e9-bd0c-f9f9394e188b-maas-api-service-ca\") pod \"8e2a04ca-f403-49e9-bd0c-f9f9394e188b\" (UID: \"8e2a04ca-f403-49e9-bd0c-f9f9394e188b\") " Oct 29 23:30:19.664489 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.664417 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e2a04ca-f403-49e9-bd0c-f9f9394e188b-maas-api-service-ca" pod "8e2a04ca-f403-49e9-bd0c-f9f9394e188b" (UID: "8e2a04ca-f403-49e9-bd0c-f9f9394e188b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:19.664875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.664852 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2a04ca-f403-49e9-bd0c-f9f9394e188b-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.825753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.825411 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2a04ca-f403-49e9-bd0c-f9f9394e188b-kube-api-access-z54mj" pod "8e2a04ca-f403-49e9-bd0c-f9f9394e188b" (UID: "8e2a04ca-f403-49e9-bd0c-f9f9394e188b"). InnerVolumeSpecName "kube-api-access-z54mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:19.872911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:19.872866 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z54mj\" (UniqueName: \"kubernetes.io/projected/8e2a04ca-f403-49e9-bd0c-f9f9394e188b-kube-api-access-z54mj\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:20.131607 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.131578 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g" Oct 29 23:30:20.200357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.200287 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" event={"ID":"8e2a04ca-f403-49e9-bd0c-f9f9394e188b","Type":"ContainerDied","Data":"6390790e34d44b529a938ff4120f3bc190c9d7dd47dba00a0ca84c71080bc20e"} Oct 29 23:30:20.200807 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.200324 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf" Oct 29 23:30:20.200807 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.200434 5057 scope.go:122] "RemoveContainer" containerID="18c902eafa2b5a72728196ddd7d9480448cc6cffb60617a82e1ade24bdf3f5b5" Oct 29 23:30:20.211990 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.211943 5057 generic.go:356] "Generic (PLEG): container finished" podID="a2de74b6-1d73-4ab4-adaa-e081fbc0240d" containerID="2dadea59d9b84f953c184dbd3269857d753ee365301f659e3e74e75b1f23c7bd" exitCode=60 Oct 29 23:30:20.212174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.212081 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" event={"ID":"a2de74b6-1d73-4ab4-adaa-e081fbc0240d","Type":"ContainerDied","Data":"2dadea59d9b84f953c184dbd3269857d753ee365301f659e3e74e75b1f23c7bd"} Oct 29 23:30:20.212788 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.212646 5057 scope.go:122] "RemoveContainer" containerID="2dadea59d9b84f953c184dbd3269857d753ee365301f659e3e74e75b1f23c7bd" Oct 29 23:30:20.213770 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:20.213147 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs_odh-ai-gateway-infra(a2de74b6-1d73-4ab4-adaa-e081fbc0240d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" podUID="a2de74b6-1d73-4ab4-adaa-e081fbc0240d" Oct 29 23:30:20.216455 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.216430 5057 generic.go:356] "Generic (PLEG): container finished" podID="eea3ec3f-906d-4185-b80b-00c0f50bee8e" containerID="57804e426c88ddaa11e98272141dfea13a63b477dac87dd5313cc68ca88c7558" exitCode=60 Oct 29 23:30:20.216575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.216506 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd" event={"ID":"eea3ec3f-906d-4185-b80b-00c0f50bee8e","Type":"ContainerDied","Data":"57804e426c88ddaa11e98272141dfea13a63b477dac87dd5313cc68ca88c7558"} Oct 29 23:30:20.217242 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.217222 5057 scope.go:122] "RemoveContainer" containerID="aca325121243eb93c173ddb4ce63b22ae90843646962bc6970cb95fd25c963cf" Oct 29 23:30:20.217347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.217258 5057 scope.go:122] "RemoveContainer" containerID="57804e426c88ddaa11e98272141dfea13a63b477dac87dd5313cc68ca88c7558" Oct 29 23:30:20.217781 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:20.217725 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd_odh-ai-gateway-infra(eea3ec3f-906d-4185-b80b-00c0f50bee8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd" podUID="eea3ec3f-906d-4185-b80b-00c0f50bee8e" Oct 29 23:30:20.221204 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.221155 5057 generic.go:356] "Generic (PLEG): container finished" podID="595f0d6b-00c5-4a95-8881-8191e7de38ff" containerID="9346031c15b404ed0266dc2aeed2ca00d94f9fecbfa80e1358dddf273fb39d22" exitCode=60 Oct 29 23:30:20.221340 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.221317 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" event={"ID":"595f0d6b-00c5-4a95-8881-8191e7de38ff","Type":"ContainerDied","Data":"9346031c15b404ed0266dc2aeed2ca00d94f9fecbfa80e1358dddf273fb39d22"} Oct 29 23:30:20.221753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.221735 5057 scope.go:122] "RemoveContainer" containerID="9346031c15b404ed0266dc2aeed2ca00d94f9fecbfa80e1358dddf273fb39d22" Oct 29 23:30:20.222114 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:20.222090 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh_odh-ai-gateway-infra(595f0d6b-00c5-4a95-8881-8191e7de38ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" podUID="595f0d6b-00c5-4a95-8881-8191e7de38ff" Oct 29 23:30:20.226441 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.226417 5057 generic.go:356] "Generic (PLEG): container finished" podID="664c869e-4b17-41bb-946b-d2ead80eff08" containerID="2d0e94fb5de1ec846ba4f5a26f36c33149f7dddd6c9882196690889048f7c02a" exitCode=60 Oct 29 23:30:20.226676 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.226488 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" event={"ID":"664c869e-4b17-41bb-946b-d2ead80eff08","Type":"ContainerDied","Data":"2d0e94fb5de1ec846ba4f5a26f36c33149f7dddd6c9882196690889048f7c02a"} Oct 29 23:30:20.227170 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.227138 5057 scope.go:122] "RemoveContainer" containerID="2d0e94fb5de1ec846ba4f5a26f36c33149f7dddd6c9882196690889048f7c02a" Oct 29 23:30:20.227632 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:20.227599 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp_odh-ai-gateway-infra(664c869e-4b17-41bb-946b-d2ead80eff08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" podUID="664c869e-4b17-41bb-946b-d2ead80eff08" Oct 29 23:30:20.229009 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.228985 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g" event={"ID":"04c2e336-3da0-4816-8313-e8fdeb0edb5f","Type":"ContainerDied","Data":"0364c21e1eb1c93d613aee99515182ab54da320d8dd103ef0fd9a577060d7658"} Oct 29 23:30:20.229095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.229071 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g" Oct 29 23:30:20.240552 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.240435 5057 scope.go:122] "RemoveContainer" containerID="12369f5bb231ea4710d80139932fa7beb60b0afd6f276fbfe48e64b3c0cde59a" Oct 29 23:30:20.264657 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.264633 5057 scope.go:122] "RemoveContainer" containerID="9a05eec2ebc30904e1601967da99cbd78c985f58dfb1b28aac64dd640f09c3df" Oct 29 23:30:20.268341 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.268291 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf"] Oct 29 23:30:20.274939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.274624 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888610-9gbgf"] Oct 29 23:30:20.285421 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.281820 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c2e336-3da0-4816-8313-e8fdeb0edb5f-kube-api-access-wdv9m\" (UniqueName: \"kubernetes.io/projected/04c2e336-3da0-4816-8313-e8fdeb0edb5f-kube-api-access-wdv9m\") pod \"04c2e336-3da0-4816-8313-e8fdeb0edb5f\" (UID: \"04c2e336-3da0-4816-8313-e8fdeb0edb5f\") " Oct 29 23:30:20.285421 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.282637 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04c2e336-3da0-4816-8313-e8fdeb0edb5f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04c2e336-3da0-4816-8313-e8fdeb0edb5f-maas-api-service-ca\") pod \"04c2e336-3da0-4816-8313-e8fdeb0edb5f\" (UID: \"04c2e336-3da0-4816-8313-e8fdeb0edb5f\") " Oct 29 23:30:20.285421 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.284968 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c2e336-3da0-4816-8313-e8fdeb0edb5f-maas-api-service-ca" pod "04c2e336-3da0-4816-8313-e8fdeb0edb5f" (UID: "04c2e336-3da0-4816-8313-e8fdeb0edb5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:20.326763 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.326676 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c2e336-3da0-4816-8313-e8fdeb0edb5f-kube-api-access-wdv9m" pod "04c2e336-3da0-4816-8313-e8fdeb0edb5f" (UID: "04c2e336-3da0-4816-8313-e8fdeb0edb5f"). InnerVolumeSpecName "kube-api-access-wdv9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:20.457399 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.448542 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdv9m\" (UniqueName: \"kubernetes.io/projected/04c2e336-3da0-4816-8313-e8fdeb0edb5f-kube-api-access-wdv9m\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:20.457399 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.448572 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04c2e336-3da0-4816-8313-e8fdeb0edb5f-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:20.574076 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.477814 5057 scope.go:122] "RemoveContainer" containerID="c928a17b85d6c7f973512f4896d844c1df3e3981f355b445b60be9afef1fa81a" Oct 29 23:30:20.676342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:20.669231 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh"] Oct 29 23:30:21.139377 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:21.139299 5057 scope.go:122] "RemoveContainer" containerID="28d07885a76f1ce9bfe946a83e8d8dcf60d7fb8b868ad42f573fec693321a603" Oct 29 23:30:21.153409 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:21.153377 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp"] Oct 29 23:30:21.182766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:21.172691 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs"] Oct 29 23:30:21.195316 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:21.195291 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd"] Oct 29 23:30:21.220947 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:21.220871 5057 scope.go:122] "RemoveContainer" containerID="ed3f6319454b549319e0da0c79b4adec23048ec7e39839d96e1c9892b3d62b4e" Oct 29 23:30:21.280564 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:21.280531 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" Oct 29 23:30:21.285170 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:21.285142 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6" event={"ID":"13000cb3-436f-4fec-bb70-7e09a449dec0","Type":"ContainerStarted","Data":"d0e9add1f023eee409dcba0071c436016f4eece10c45f505907306eb3b923cb7"} Oct 29 23:30:21.290141 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:21.290111 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8" event={"ID":"e2f3e7bb-6ca1-46a8-a146-cb7c5690f377","Type":"ContainerStarted","Data":"d88a641fcc3d527da3b982b2c9ed00d5e26c4cdb7bd8ca9dcda2376da772f364"} Oct 29 23:30:21.291859 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:21.291836 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" event={"ID":"9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8","Type":"ContainerDied","Data":"e6704358d51a52ef8d2fa680e7aba0ba97ffebe70adc7a6f005030921b6e715a"} Oct 29 23:30:21.291995 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:21.291864 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z" Oct 29 23:30:21.295068 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:21.295041 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns" event={"ID":"cecae6fc-a464-491b-945e-7344fca3eacd","Type":"ContainerStarted","Data":"cc499e45ca3adb361ee7c5db129fa2ce62a264035eaf14da67a10b6c2c743c4d"} Oct 29 23:30:21.295564 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:21.295545 5057 scope.go:122] "RemoveContainer" containerID="cc499e45ca3adb361ee7c5db129fa2ce62a264035eaf14da67a10b6c2c743c4d" Oct 29 23:30:21.295959 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:21.295937 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns_odh-ai-gateway-infra(cecae6fc-a464-491b-945e-7344fca3eacd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns" podUID="cecae6fc-a464-491b-945e-7344fca3eacd" Oct 29 23:30:21.306982 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:21.306204 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" event={"ID":"7eb091cb-8d53-4b37-835b-591edcba96f1","Type":"ContainerStarted","Data":"c0d94f49702de604255c5625089482f39d4ebab34c89bf50108df604aff21353"} Oct 29 23:30:21.306982 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:21.306640 5057 scope.go:122] "RemoveContainer" containerID="c0d94f49702de604255c5625089482f39d4ebab34c89bf50108df604aff21353" Oct 29 23:30:21.306982 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:21.306927 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6_odh-ai-gateway-infra(7eb091cb-8d53-4b37-835b-591edcba96f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" podUID="7eb091cb-8d53-4b37-835b-591edcba96f1" Oct 29 23:30:21.388245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:21.352798 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8-kube-api-access-ddknw\" (UniqueName: \"kubernetes.io/projected/9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8-kube-api-access-ddknw\") pod \"9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8\" (UID: \"9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8\") " Oct 29 23:30:21.388245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:21.352895 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8-maas-api-service-ca\") pod \"9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8\" (UID: \"9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8\") " Oct 29 23:30:21.388245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:21.355796 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8-maas-api-service-ca" pod "9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8" (UID: "9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:21.388245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:21.360508 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8-kube-api-access-ddknw" pod "9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8" (UID: "9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8"). InnerVolumeSpecName "kube-api-access-ddknw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:21.516363 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:21.466364 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddknw\" (UniqueName: \"kubernetes.io/projected/9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8-kube-api-access-ddknw\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:21.516363 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:21.466406 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:22.085844 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.085779 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6" podStartSLOduration=22.085759127 podStartE2EDuration="22.085759127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:21.790826494 +0000 UTC m=+2868.442325036" watchObservedRunningTime="2026-10-29 23:30:22.085759127 +0000 UTC m=+2868.737257658" Oct 29 23:30:22.107270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.104718 5057 scope.go:122] "RemoveContainer" containerID="1d4ee90d88bf7f72a2cedb8969db35e175fbe13ac39dfa635f110a0aec6a7903" Oct 29 23:30:22.145222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.139391 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" Oct 29 23:30:22.212922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.210914 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e2a04ca-f403-49e9-bd0c-f9f9394e188b" path="/var/lib/kubelet/pods/8e2a04ca-f403-49e9-bd0c-f9f9394e188b/volumes" Oct 29 23:30:22.212922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.212772 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g"] Oct 29 23:30:22.212922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.212810 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888610-h4m6g"] Oct 29 23:30:22.259257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.258358 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c2a4a82-5ab9-4c15-8a1d-54da076ff3db-kube-api-access-ftkqt\" (UniqueName: \"kubernetes.io/projected/1c2a4a82-5ab9-4c15-8a1d-54da076ff3db-kube-api-access-ftkqt\") pod \"1c2a4a82-5ab9-4c15-8a1d-54da076ff3db\" (UID: \"1c2a4a82-5ab9-4c15-8a1d-54da076ff3db\") " Oct 29 23:30:22.259257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.258628 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c2a4a82-5ab9-4c15-8a1d-54da076ff3db-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2a4a82-5ab9-4c15-8a1d-54da076ff3db-maas-api-service-ca\") pod \"1c2a4a82-5ab9-4c15-8a1d-54da076ff3db\" (UID: \"1c2a4a82-5ab9-4c15-8a1d-54da076ff3db\") " Oct 29 23:30:22.267305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.267259 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c2a4a82-5ab9-4c15-8a1d-54da076ff3db-maas-api-service-ca" pod "1c2a4a82-5ab9-4c15-8a1d-54da076ff3db" (UID: "1c2a4a82-5ab9-4c15-8a1d-54da076ff3db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:22.267981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.267616 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2a4a82-5ab9-4c15-8a1d-54da076ff3db-kube-api-access-ftkqt" pod "1c2a4a82-5ab9-4c15-8a1d-54da076ff3db" (UID: "1c2a4a82-5ab9-4c15-8a1d-54da076ff3db"). InnerVolumeSpecName "kube-api-access-ftkqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:22.347339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.347253 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" Oct 29 23:30:22.353048 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.353010 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" event={"ID":"1c2a4a82-5ab9-4c15-8a1d-54da076ff3db","Type":"ContainerDied","Data":"4984bf0a9b99ff27f8d64f02e5a555bad8636b6ee09620ae3a30ca581dd64fdf"} Oct 29 23:30:22.353245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.353083 5057 scope.go:122] "RemoveContainer" containerID="c576f905edb9017c8c463b76dad68e799fd283bb37f662a761b54dd22fb65924" Oct 29 23:30:22.353245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.353235 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w" Oct 29 23:30:22.394210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.382617 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftkqt\" (UniqueName: \"kubernetes.io/projected/1c2a4a82-5ab9-4c15-8a1d-54da076ff3db-kube-api-access-ftkqt\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:22.394210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.382658 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2a4a82-5ab9-4c15-8a1d-54da076ff3db-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:22.569256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.555050 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2de74b6-1d73-4ab4-adaa-e081fbc0240d-kube-api-access-f2425\" (UniqueName: \"kubernetes.io/projected/a2de74b6-1d73-4ab4-adaa-e081fbc0240d-kube-api-access-f2425\") pod \"a2de74b6-1d73-4ab4-adaa-e081fbc0240d\" (UID: \"a2de74b6-1d73-4ab4-adaa-e081fbc0240d\") " Oct 29 23:30:22.569695 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.569641 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2de74b6-1d73-4ab4-adaa-e081fbc0240d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2de74b6-1d73-4ab4-adaa-e081fbc0240d-maas-api-service-ca\") pod \"a2de74b6-1d73-4ab4-adaa-e081fbc0240d\" (UID: \"a2de74b6-1d73-4ab4-adaa-e081fbc0240d\") " Oct 29 23:30:22.623263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.612423 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2de74b6-1d73-4ab4-adaa-e081fbc0240d-maas-api-service-ca" pod "a2de74b6-1d73-4ab4-adaa-e081fbc0240d" (UID: "a2de74b6-1d73-4ab4-adaa-e081fbc0240d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:22.648374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.648336 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv" event={"ID":"77571d53-cd5a-4b79-922b-3884bf5f7fcb","Type":"ContainerStarted","Data":"b5c6d4ef186aafcbacb107fdb3569b238bb7a4621b4dc0b6d9e89cb67018ab57"} Oct 29 23:30:22.652476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.652419 5057 generic.go:356] "Generic (PLEG): container finished" podID="13000cb3-436f-4fec-bb70-7e09a449dec0" containerID="d0e9add1f023eee409dcba0071c436016f4eece10c45f505907306eb3b923cb7" exitCode=60 Oct 29 23:30:22.652813 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.652777 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6" event={"ID":"13000cb3-436f-4fec-bb70-7e09a449dec0","Type":"ContainerDied","Data":"d0e9add1f023eee409dcba0071c436016f4eece10c45f505907306eb3b923cb7"} Oct 29 23:30:22.669464 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.669436 5057 scope.go:122] "RemoveContainer" containerID="d0e9add1f023eee409dcba0071c436016f4eece10c45f505907306eb3b923cb7" Oct 29 23:30:22.669958 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:22.669930 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6_odh-ai-gateway-infra(13000cb3-436f-4fec-bb70-7e09a449dec0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6" podUID="13000cb3-436f-4fec-bb70-7e09a449dec0" Oct 29 23:30:22.677625 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.677601 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2de74b6-1d73-4ab4-adaa-e081fbc0240d-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:22.701915 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.701734 5057 generic.go:356] "Generic (PLEG): container finished" podID="e2f3e7bb-6ca1-46a8-a146-cb7c5690f377" containerID="d88a641fcc3d527da3b982b2c9ed00d5e26c4cdb7bd8ca9dcda2376da772f364" exitCode=60 Oct 29 23:30:22.702702 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.702670 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8" event={"ID":"e2f3e7bb-6ca1-46a8-a146-cb7c5690f377","Type":"ContainerDied","Data":"d88a641fcc3d527da3b982b2c9ed00d5e26c4cdb7bd8ca9dcda2376da772f364"} Oct 29 23:30:22.931414 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:22.931379 5057 scope.go:122] "RemoveContainer" containerID="d88a641fcc3d527da3b982b2c9ed00d5e26c4cdb7bd8ca9dcda2376da772f364" Oct 29 23:30:22.933103 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:22.933066 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8_odh-ai-gateway-infra(e2f3e7bb-6ca1-46a8-a146-cb7c5690f377)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8" podUID="e2f3e7bb-6ca1-46a8-a146-cb7c5690f377" Oct 29 23:30:23.486208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:23.480114 5057 scope.go:122] "RemoveContainer" containerID="b7a221f714d8db67ee8565967c2658d1df4cd3a00bd533fd7c6efadd9e2f5766" Oct 29 23:30:23.610221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:23.610116 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2de74b6-1d73-4ab4-adaa-e081fbc0240d-kube-api-access-f2425" pod "a2de74b6-1d73-4ab4-adaa-e081fbc0240d" (UID: "a2de74b6-1d73-4ab4-adaa-e081fbc0240d"). InnerVolumeSpecName "kube-api-access-f2425". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:23.920341 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:23.881424 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2qkxv" podUID="907c36dd-72ba-42b8-86f1-9ff84400b4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 29 23:30:24.106088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.106051 5057 scope.go:122] "RemoveContainer" containerID="c08ffcca76645ba20f222b0d9d7e5b663f9aae99e67958a89cdb1307a985ba3d" Oct 29 23:30:24.109155 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:23.925543 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:30:24.127413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.032310 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:30:24.134144 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.134111 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" Oct 29 23:30:24.159524 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.159491 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" Oct 29 23:30:24.176549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.174899 5057 generic.go:356] "Generic (PLEG): container finished" podID="cecae6fc-a464-491b-945e-7344fca3eacd" containerID="cc499e45ca3adb361ee7c5db129fa2ce62a264035eaf14da67a10b6c2c743c4d" exitCode=60 Oct 29 23:30:24.185657 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.185620 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" Oct 29 23:30:24.188388 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.188355 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd" Oct 29 23:30:24.194113 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.194087 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2de74b6-1d73-4ab4-adaa-e081fbc0240d-kube-api-access-f2425\" (UniqueName: \"kubernetes.io/projected/a2de74b6-1d73-4ab4-adaa-e081fbc0240d-kube-api-access-f2425\") pod \"a2de74b6-1d73-4ab4-adaa-e081fbc0240d\" (UID: \"a2de74b6-1d73-4ab4-adaa-e081fbc0240d\") " Oct 29 23:30:24.198107 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:30:24.195783 5057 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/a2de74b6-1d73-4ab4-adaa-e081fbc0240d/volumes/kubernetes.io~projected/kube-api-access-f2425 Oct 29 23:30:24.198265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.198175 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2de74b6-1d73-4ab4-adaa-e081fbc0240d-kube-api-access-f2425" pod "a2de74b6-1d73-4ab4-adaa-e081fbc0240d" (UID: "a2de74b6-1d73-4ab4-adaa-e081fbc0240d"). InnerVolumeSpecName "kube-api-access-f2425". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:24.200382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.200338 5057 generic.go:356] "Generic (PLEG): container finished" podID="7eb091cb-8d53-4b37-835b-591edcba96f1" containerID="c0d94f49702de604255c5625089482f39d4ebab34c89bf50108df604aff21353" exitCode=60 Oct 29 23:30:24.207211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.202844 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/664c869e-4b17-41bb-946b-d2ead80eff08-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/664c869e-4b17-41bb-946b-d2ead80eff08-maas-api-service-ca\") pod \"664c869e-4b17-41bb-946b-d2ead80eff08\" (UID: \"664c869e-4b17-41bb-946b-d2ead80eff08\") " Oct 29 23:30:24.207211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.202886 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/664c869e-4b17-41bb-946b-d2ead80eff08-kube-api-access-9ctvb\" (UniqueName: \"kubernetes.io/projected/664c869e-4b17-41bb-946b-d2ead80eff08-kube-api-access-9ctvb\") pod \"664c869e-4b17-41bb-946b-d2ead80eff08\" (UID: \"664c869e-4b17-41bb-946b-d2ead80eff08\") " Oct 29 23:30:24.207211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.203284 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/664c869e-4b17-41bb-946b-d2ead80eff08-maas-api-service-ca" pod "664c869e-4b17-41bb-946b-d2ead80eff08" (UID: "664c869e-4b17-41bb-946b-d2ead80eff08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:24.209244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.208176 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04c2e336-3da0-4816-8313-e8fdeb0edb5f" path="/var/lib/kubelet/pods/04c2e336-3da0-4816-8313-e8fdeb0edb5f/volumes" Oct 29 23:30:24.209244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.208577 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/664c869e-4b17-41bb-946b-d2ead80eff08-kube-api-access-9ctvb" pod "664c869e-4b17-41bb-946b-d2ead80eff08" (UID: "664c869e-4b17-41bb-946b-d2ead80eff08"). InnerVolumeSpecName "kube-api-access-9ctvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:24.209490 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.209473 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2425\" (UniqueName: \"kubernetes.io/projected/a2de74b6-1d73-4ab4-adaa-e081fbc0240d-kube-api-access-f2425\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:24.209545 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.209498 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/664c869e-4b17-41bb-946b-d2ead80eff08-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:24.247023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.246985 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns" event={"ID":"cecae6fc-a464-491b-945e-7344fca3eacd","Type":"ContainerDied","Data":"cc499e45ca3adb361ee7c5db129fa2ce62a264035eaf14da67a10b6c2c743c4d"} Oct 29 23:30:24.247166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.247066 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6"] Oct 29 23:30:24.247166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.247103 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs" event={"ID":"a2de74b6-1d73-4ab4-adaa-e081fbc0240d","Type":"ContainerDied","Data":"88d29f14613ec6d9d003e61f32e588fb0e7fc3d3089bc91ae193acca2209cd46"} Oct 29 23:30:24.247166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.247121 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w"] Oct 29 23:30:24.247166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.247142 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888610-k686w"] Oct 29 23:30:24.247166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.247165 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" event={"ID":"7eb091cb-8d53-4b37-835b-591edcba96f1","Type":"ContainerDied","Data":"c0d94f49702de604255c5625089482f39d4ebab34c89bf50108df604aff21353"} Oct 29 23:30:24.247477 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.247243 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z"] Oct 29 23:30:24.247477 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.247265 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888610-vmk9z"] Oct 29 23:30:24.247477 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.247284 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8"] Oct 29 23:30:24.247477 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.247283 5057 scope.go:122] "RemoveContainer" containerID="cc41d0ce57cace3d0c30b3f6ebb5e57b32dcc15124628f16eeb426aa6481d9ea" Oct 29 23:30:24.247803 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.247784 5057 scope.go:122] "RemoveContainer" containerID="c0d94f49702de604255c5625089482f39d4ebab34c89bf50108df604aff21353" Oct 29 23:30:24.248199 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.248163 5057 scope.go:122] "RemoveContainer" containerID="cc499e45ca3adb361ee7c5db129fa2ce62a264035eaf14da67a10b6c2c743c4d" Oct 29 23:30:24.248432 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:24.248410 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6_odh-ai-gateway-infra(7eb091cb-8d53-4b37-835b-591edcba96f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" podUID="7eb091cb-8d53-4b37-835b-591edcba96f1" Oct 29 23:30:24.248541 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:24.248515 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns_odh-ai-gateway-infra(cecae6fc-a464-491b-945e-7344fca3eacd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns" podUID="cecae6fc-a464-491b-945e-7344fca3eacd" Oct 29 23:30:24.252822 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.252798 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6"] Oct 29 23:30:24.266527 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.266498 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns"] Oct 29 23:30:24.273248 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.273230 5057 scope.go:122] "RemoveContainer" containerID="2dadea59d9b84f953c184dbd3269857d753ee365301f659e3e74e75b1f23c7bd" Oct 29 23:30:24.285639 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.285619 5057 scope.go:122] "RemoveContainer" containerID="99ed2333cace3765aaf5c55d499e51b3b4cb6900dc95bf0c480e425542837dea" Oct 29 23:30:24.310179 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.310147 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/595f0d6b-00c5-4a95-8881-8191e7de38ff-kube-api-access-95wd9\" (UniqueName: \"kubernetes.io/projected/595f0d6b-00c5-4a95-8881-8191e7de38ff-kube-api-access-95wd9\") pod \"595f0d6b-00c5-4a95-8881-8191e7de38ff\" (UID: \"595f0d6b-00c5-4a95-8881-8191e7de38ff\") " Oct 29 23:30:24.310507 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.310390 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea3ec3f-906d-4185-b80b-00c0f50bee8e-kube-api-access-2sgb8\" (UniqueName: \"kubernetes.io/projected/eea3ec3f-906d-4185-b80b-00c0f50bee8e-kube-api-access-2sgb8\") pod \"eea3ec3f-906d-4185-b80b-00c0f50bee8e\" (UID: \"eea3ec3f-906d-4185-b80b-00c0f50bee8e\") " Oct 29 23:30:24.310507 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.310430 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea3ec3f-906d-4185-b80b-00c0f50bee8e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea3ec3f-906d-4185-b80b-00c0f50bee8e-maas-api-service-ca\") pod \"eea3ec3f-906d-4185-b80b-00c0f50bee8e\" (UID: \"eea3ec3f-906d-4185-b80b-00c0f50bee8e\") " Oct 29 23:30:24.310745 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.310523 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/595f0d6b-00c5-4a95-8881-8191e7de38ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/595f0d6b-00c5-4a95-8881-8191e7de38ff-maas-api-service-ca\") pod \"595f0d6b-00c5-4a95-8881-8191e7de38ff\" (UID: \"595f0d6b-00c5-4a95-8881-8191e7de38ff\") " Oct 29 23:30:24.310906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.310883 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/595f0d6b-00c5-4a95-8881-8191e7de38ff-maas-api-service-ca" pod "595f0d6b-00c5-4a95-8881-8191e7de38ff" (UID: "595f0d6b-00c5-4a95-8881-8191e7de38ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:24.311893 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.311118 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea3ec3f-906d-4185-b80b-00c0f50bee8e-maas-api-service-ca" pod "eea3ec3f-906d-4185-b80b-00c0f50bee8e" (UID: "eea3ec3f-906d-4185-b80b-00c0f50bee8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:24.312077 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.311874 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/595f0d6b-00c5-4a95-8881-8191e7de38ff-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:24.312172 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.312089 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ctvb\" (UniqueName: \"kubernetes.io/projected/664c869e-4b17-41bb-946b-d2ead80eff08-kube-api-access-9ctvb\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:24.313433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.313407 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595f0d6b-00c5-4a95-8881-8191e7de38ff-kube-api-access-95wd9" pod "595f0d6b-00c5-4a95-8881-8191e7de38ff" (UID: "595f0d6b-00c5-4a95-8881-8191e7de38ff"). InnerVolumeSpecName "kube-api-access-95wd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:24.313542 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.313462 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea3ec3f-906d-4185-b80b-00c0f50bee8e-kube-api-access-2sgb8" pod "eea3ec3f-906d-4185-b80b-00c0f50bee8e" (UID: "eea3ec3f-906d-4185-b80b-00c0f50bee8e"). InnerVolumeSpecName "kube-api-access-2sgb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:24.370458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.370400 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv" podStartSLOduration=24.370383271 podStartE2EDuration="24.370383271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:24.362789849 +0000 UTC m=+2871.014288379" watchObservedRunningTime="2026-10-29 23:30:24.370383271 +0000 UTC m=+2871.021881800" Oct 29 23:30:24.412799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.412760 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs"] Oct 29 23:30:24.414545 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.414509 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95wd9\" (UniqueName: \"kubernetes.io/projected/595f0d6b-00c5-4a95-8881-8191e7de38ff-kube-api-access-95wd9\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:24.414545 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.414536 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sgb8\" (UniqueName: \"kubernetes.io/projected/eea3ec3f-906d-4185-b80b-00c0f50bee8e-kube-api-access-2sgb8\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:24.414545 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.414558 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea3ec3f-906d-4185-b80b-00c0f50bee8e-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:24.431468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:24.431363 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888610-2zjcs"] Oct 29 23:30:25.213153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.213119 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" Oct 29 23:30:25.213153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.213140 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp" event={"ID":"664c869e-4b17-41bb-946b-d2ead80eff08","Type":"ContainerDied","Data":"370195e75e9ebcbe8f25989fb6aabcce7d64ddfb48677181d029af955e20f57b"} Oct 29 23:30:25.213740 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.213219 5057 scope.go:122] "RemoveContainer" containerID="2d0e94fb5de1ec846ba4f5a26f36c33149f7dddd6c9882196690889048f7c02a" Oct 29 23:30:25.215858 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.215833 5057 generic.go:356] "Generic (PLEG): container finished" podID="77571d53-cd5a-4b79-922b-3884bf5f7fcb" containerID="b5c6d4ef186aafcbacb107fdb3569b238bb7a4621b4dc0b6d9e89cb67018ab57" exitCode=60 Oct 29 23:30:25.216266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.216013 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv" event={"ID":"77571d53-cd5a-4b79-922b-3884bf5f7fcb","Type":"ContainerDied","Data":"b5c6d4ef186aafcbacb107fdb3569b238bb7a4621b4dc0b6d9e89cb67018ab57"} Oct 29 23:30:25.216997 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.216827 5057 scope.go:122] "RemoveContainer" containerID="b5c6d4ef186aafcbacb107fdb3569b238bb7a4621b4dc0b6d9e89cb67018ab57" Oct 29 23:30:25.217297 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:25.217272 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv_odh-ai-gateway-infra(77571d53-cd5a-4b79-922b-3884bf5f7fcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv" podUID="77571d53-cd5a-4b79-922b-3884bf5f7fcb" Oct 29 23:30:25.227251 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.227216 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd" event={"ID":"eea3ec3f-906d-4185-b80b-00c0f50bee8e","Type":"ContainerDied","Data":"540e165c81964e429301998d2b7404786bbd8c411dcee5d9a9e10c319101cf02"} Oct 29 23:30:25.227377 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.227317 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd" Oct 29 23:30:25.234594 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.234368 5057 scope.go:122] "RemoveContainer" containerID="c928a17b85d6c7f973512f4896d844c1df3e3981f355b445b60be9afef1fa81a" Oct 29 23:30:25.271453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.270683 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" event={"ID":"595f0d6b-00c5-4a95-8881-8191e7de38ff","Type":"ContainerDied","Data":"48a72e7212e707826615698215b657f7795e9fcc06b1798590f2a68692b356ea"} Oct 29 23:30:25.271453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.270803 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh" Oct 29 23:30:25.361954 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.361931 5057 scope.go:122] "RemoveContainer" containerID="57804e426c88ddaa11e98272141dfea13a63b477dac87dd5313cc68ca88c7558" Oct 29 23:30:25.373321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.373283 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp"] Oct 29 23:30:25.383460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.383404 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888610-kxcvp"] Oct 29 23:30:25.394642 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.394604 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv"] Oct 29 23:30:25.397049 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.397020 5057 scope.go:122] "RemoveContainer" containerID="9346031c15b404ed0266dc2aeed2ca00d94f9fecbfa80e1358dddf273fb39d22" Oct 29 23:30:25.599831 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.599796 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c2a4a82-5ab9-4c15-8a1d-54da076ff3db" path="/var/lib/kubelet/pods/1c2a4a82-5ab9-4c15-8a1d-54da076ff3db/volumes" Oct 29 23:30:25.601752 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.601724 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="664c869e-4b17-41bb-946b-d2ead80eff08" path="/var/lib/kubelet/pods/664c869e-4b17-41bb-946b-d2ead80eff08/volumes" Oct 29 23:30:25.603263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.603244 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8" path="/var/lib/kubelet/pods/9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8/volumes" Oct 29 23:30:25.604766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.604750 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2de74b6-1d73-4ab4-adaa-e081fbc0240d" path="/var/lib/kubelet/pods/a2de74b6-1d73-4ab4-adaa-e081fbc0240d/volumes" Oct 29 23:30:25.640445 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.640412 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd"] Oct 29 23:30:25.652601 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.652552 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888610-6k9fd"] Oct 29 23:30:25.684987 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.684961 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8" Oct 29 23:30:25.698283 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.698248 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh"] Oct 29 23:30:25.706476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.706415 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888610-ct6rh"] Oct 29 23:30:25.763168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.763136 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6" Oct 29 23:30:25.858449 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.857950 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2f3e7bb-6ca1-46a8-a146-cb7c5690f377-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f3e7bb-6ca1-46a8-a146-cb7c5690f377-maas-api-service-ca\") pod \"e2f3e7bb-6ca1-46a8-a146-cb7c5690f377\" (UID: \"e2f3e7bb-6ca1-46a8-a146-cb7c5690f377\") " Oct 29 23:30:25.863891 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.863860 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f3e7bb-6ca1-46a8-a146-cb7c5690f377-kube-api-access-cpdfx\" (UniqueName: \"kubernetes.io/projected/e2f3e7bb-6ca1-46a8-a146-cb7c5690f377-kube-api-access-cpdfx\") pod \"e2f3e7bb-6ca1-46a8-a146-cb7c5690f377\" (UID: \"e2f3e7bb-6ca1-46a8-a146-cb7c5690f377\") " Oct 29 23:30:25.868384 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.868331 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f3e7bb-6ca1-46a8-a146-cb7c5690f377-maas-api-service-ca" pod "e2f3e7bb-6ca1-46a8-a146-cb7c5690f377" (UID: "e2f3e7bb-6ca1-46a8-a146-cb7c5690f377"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:25.876524 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.876501 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f3e7bb-6ca1-46a8-a146-cb7c5690f377-kube-api-access-cpdfx" pod "e2f3e7bb-6ca1-46a8-a146-cb7c5690f377" (UID: "e2f3e7bb-6ca1-46a8-a146-cb7c5690f377"). InnerVolumeSpecName "kube-api-access-cpdfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:25.890350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.890314 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns" Oct 29 23:30:25.898362 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.898338 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" Oct 29 23:30:25.966675 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.966634 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13000cb3-436f-4fec-bb70-7e09a449dec0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13000cb3-436f-4fec-bb70-7e09a449dec0-maas-api-service-ca\") pod \"13000cb3-436f-4fec-bb70-7e09a449dec0\" (UID: \"13000cb3-436f-4fec-bb70-7e09a449dec0\") " Oct 29 23:30:25.967050 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.967026 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13000cb3-436f-4fec-bb70-7e09a449dec0-kube-api-access-mp9p9\" (UniqueName: \"kubernetes.io/projected/13000cb3-436f-4fec-bb70-7e09a449dec0-kube-api-access-mp9p9\") pod \"13000cb3-436f-4fec-bb70-7e09a449dec0\" (UID: \"13000cb3-436f-4fec-bb70-7e09a449dec0\") " Oct 29 23:30:25.967225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.967033 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13000cb3-436f-4fec-bb70-7e09a449dec0-maas-api-service-ca" pod "13000cb3-436f-4fec-bb70-7e09a449dec0" (UID: "13000cb3-436f-4fec-bb70-7e09a449dec0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:25.967737 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.967711 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpdfx\" (UniqueName: \"kubernetes.io/projected/e2f3e7bb-6ca1-46a8-a146-cb7c5690f377-kube-api-access-cpdfx\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:25.967737 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.967734 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13000cb3-436f-4fec-bb70-7e09a449dec0-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:25.967990 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.967745 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f3e7bb-6ca1-46a8-a146-cb7c5690f377-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:25.969673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:25.969653 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13000cb3-436f-4fec-bb70-7e09a449dec0-kube-api-access-mp9p9" pod "13000cb3-436f-4fec-bb70-7e09a449dec0" (UID: "13000cb3-436f-4fec-bb70-7e09a449dec0"). InnerVolumeSpecName "kube-api-access-mp9p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:26.068626 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.068592 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cecae6fc-a464-491b-945e-7344fca3eacd-kube-api-access-zlzhc\" (UniqueName: \"kubernetes.io/projected/cecae6fc-a464-491b-945e-7344fca3eacd-kube-api-access-zlzhc\") pod \"cecae6fc-a464-491b-945e-7344fca3eacd\" (UID: \"cecae6fc-a464-491b-945e-7344fca3eacd\") " Oct 29 23:30:26.068805 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.068649 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eb091cb-8d53-4b37-835b-591edcba96f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb091cb-8d53-4b37-835b-591edcba96f1-maas-api-service-ca\") pod \"7eb091cb-8d53-4b37-835b-591edcba96f1\" (UID: \"7eb091cb-8d53-4b37-835b-591edcba96f1\") " Oct 29 23:30:26.068805 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.068761 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eb091cb-8d53-4b37-835b-591edcba96f1-kube-api-access-stghw\" (UniqueName: \"kubernetes.io/projected/7eb091cb-8d53-4b37-835b-591edcba96f1-kube-api-access-stghw\") pod \"7eb091cb-8d53-4b37-835b-591edcba96f1\" (UID: \"7eb091cb-8d53-4b37-835b-591edcba96f1\") " Oct 29 23:30:26.069061 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.069037 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cecae6fc-a464-491b-945e-7344fca3eacd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cecae6fc-a464-491b-945e-7344fca3eacd-maas-api-service-ca\") pod \"cecae6fc-a464-491b-945e-7344fca3eacd\" (UID: \"cecae6fc-a464-491b-945e-7344fca3eacd\") " Oct 29 23:30:26.069250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.069066 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eb091cb-8d53-4b37-835b-591edcba96f1-maas-api-service-ca" pod "7eb091cb-8d53-4b37-835b-591edcba96f1" (UID: "7eb091cb-8d53-4b37-835b-591edcba96f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:26.069452 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.069413 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cecae6fc-a464-491b-945e-7344fca3eacd-maas-api-service-ca" pod "cecae6fc-a464-491b-945e-7344fca3eacd" (UID: "cecae6fc-a464-491b-945e-7344fca3eacd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:26.070109 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.070089 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cecae6fc-a464-491b-945e-7344fca3eacd-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:26.070249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.070112 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp9p9\" (UniqueName: \"kubernetes.io/projected/13000cb3-436f-4fec-bb70-7e09a449dec0-kube-api-access-mp9p9\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:26.070249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.070126 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb091cb-8d53-4b37-835b-591edcba96f1-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:26.071036 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.071014 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb091cb-8d53-4b37-835b-591edcba96f1-kube-api-access-stghw" pod "7eb091cb-8d53-4b37-835b-591edcba96f1" (UID: "7eb091cb-8d53-4b37-835b-591edcba96f1"). InnerVolumeSpecName "kube-api-access-stghw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:26.071217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.071168 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cecae6fc-a464-491b-945e-7344fca3eacd-kube-api-access-zlzhc" pod "cecae6fc-a464-491b-945e-7344fca3eacd" (UID: "cecae6fc-a464-491b-945e-7344fca3eacd"). InnerVolumeSpecName "kube-api-access-zlzhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:26.172679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.172595 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlzhc\" (UniqueName: \"kubernetes.io/projected/cecae6fc-a464-491b-945e-7344fca3eacd-kube-api-access-zlzhc\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:26.172679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.172633 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stghw\" (UniqueName: \"kubernetes.io/projected/7eb091cb-8d53-4b37-835b-591edcba96f1-kube-api-access-stghw\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:26.304117 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.304075 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" event={"ID":"7eb091cb-8d53-4b37-835b-591edcba96f1","Type":"ContainerDied","Data":"fc4b5eee4d3ece156be416854ab798199daacbfd6449ef7afd004b5081df1f5c"} Oct 29 23:30:26.304718 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.304694 5057 scope.go:122] "RemoveContainer" containerID="c0d94f49702de604255c5625089482f39d4ebab34c89bf50108df604aff21353" Oct 29 23:30:26.304955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.304933 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6" Oct 29 23:30:26.313854 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.313821 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6" event={"ID":"13000cb3-436f-4fec-bb70-7e09a449dec0","Type":"ContainerDied","Data":"d6f18ccc8839bb0c02401081a262487f6745c5461762432a05dbd419cc4cd293"} Oct 29 23:30:26.314394 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.314373 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6" Oct 29 23:30:26.317649 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.317622 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8" event={"ID":"e2f3e7bb-6ca1-46a8-a146-cb7c5690f377","Type":"ContainerDied","Data":"963ea0d28d5d5813ab99279a04bd0c89edd42a5932ce5a20a8c9a4188c4790c0"} Oct 29 23:30:26.317760 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.317716 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8" Oct 29 23:30:26.327234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.325257 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns" event={"ID":"cecae6fc-a464-491b-945e-7344fca3eacd","Type":"ContainerDied","Data":"01485237cca941fcbe3f2c3eaa08506618522659172120ee45f1d0de8ca56285"} Oct 29 23:30:26.327234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.325345 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns" Oct 29 23:30:26.330142 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.329652 5057 scope.go:122] "RemoveContainer" containerID="d0e9add1f023eee409dcba0071c436016f4eece10c45f505907306eb3b923cb7" Oct 29 23:30:26.345661 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.345627 5057 scope.go:122] "RemoveContainer" containerID="d88a641fcc3d527da3b982b2c9ed00d5e26c4cdb7bd8ca9dcda2376da772f364" Oct 29 23:30:26.360314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.360227 5057 scope.go:122] "RemoveContainer" containerID="cc499e45ca3adb361ee7c5db129fa2ce62a264035eaf14da67a10b6c2c743c4d" Oct 29 23:30:26.445735 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.445705 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6"] Oct 29 23:30:26.457078 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.457048 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888610-l45t6"] Oct 29 23:30:26.493918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.493890 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns"] Oct 29 23:30:26.494676 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.494657 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv" Oct 29 23:30:26.498490 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.498464 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888610-7mmns"] Oct 29 23:30:26.537291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.537257 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6"] Oct 29 23:30:26.544638 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.544609 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888610-7zgn6"] Oct 29 23:30:26.581794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.581752 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8"] Oct 29 23:30:26.590497 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.590466 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888610-cjhg8"] Oct 29 23:30:26.594508 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.594484 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77571d53-cd5a-4b79-922b-3884bf5f7fcb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77571d53-cd5a-4b79-922b-3884bf5f7fcb-maas-api-service-ca\") pod \"77571d53-cd5a-4b79-922b-3884bf5f7fcb\" (UID: \"77571d53-cd5a-4b79-922b-3884bf5f7fcb\") " Oct 29 23:30:26.594632 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.594576 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77571d53-cd5a-4b79-922b-3884bf5f7fcb-kube-api-access-x9lkh\" (UniqueName: \"kubernetes.io/projected/77571d53-cd5a-4b79-922b-3884bf5f7fcb-kube-api-access-x9lkh\") pod \"77571d53-cd5a-4b79-922b-3884bf5f7fcb\" (UID: \"77571d53-cd5a-4b79-922b-3884bf5f7fcb\") " Oct 29 23:30:26.594832 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.594812 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77571d53-cd5a-4b79-922b-3884bf5f7fcb-maas-api-service-ca" pod "77571d53-cd5a-4b79-922b-3884bf5f7fcb" (UID: "77571d53-cd5a-4b79-922b-3884bf5f7fcb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:26.595260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.595243 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77571d53-cd5a-4b79-922b-3884bf5f7fcb-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:26.596700 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.596681 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77571d53-cd5a-4b79-922b-3884bf5f7fcb-kube-api-access-x9lkh" pod "77571d53-cd5a-4b79-922b-3884bf5f7fcb" (UID: "77571d53-cd5a-4b79-922b-3884bf5f7fcb"). InnerVolumeSpecName "kube-api-access-x9lkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:26.697286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:26.697202 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9lkh\" (UniqueName: \"kubernetes.io/projected/77571d53-cd5a-4b79-922b-3884bf5f7fcb-kube-api-access-x9lkh\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:30:27.337434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:27.337402 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv" Oct 29 23:30:27.337434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:27.337417 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv" event={"ID":"77571d53-cd5a-4b79-922b-3884bf5f7fcb","Type":"ContainerDied","Data":"2510a84a120139616426519d79b24cc3faa935a806cf70d8c5eb2ecb0576f5fb"} Oct 29 23:30:27.337999 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:27.337490 5057 scope.go:122] "RemoveContainer" containerID="b5c6d4ef186aafcbacb107fdb3569b238bb7a4621b4dc0b6d9e89cb67018ab57" Oct 29 23:30:27.405738 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:27.405700 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv"] Oct 29 23:30:27.414739 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:27.414708 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888610-l9xdv"] Oct 29 23:30:27.485530 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:27.485481 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13000cb3-436f-4fec-bb70-7e09a449dec0" path="/var/lib/kubelet/pods/13000cb3-436f-4fec-bb70-7e09a449dec0/volumes" Oct 29 23:30:27.486333 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:27.486316 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="595f0d6b-00c5-4a95-8881-8191e7de38ff" path="/var/lib/kubelet/pods/595f0d6b-00c5-4a95-8881-8191e7de38ff/volumes" Oct 29 23:30:27.486826 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:27.486811 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77571d53-cd5a-4b79-922b-3884bf5f7fcb" path="/var/lib/kubelet/pods/77571d53-cd5a-4b79-922b-3884bf5f7fcb/volumes" Oct 29 23:30:27.487615 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:27.487599 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eb091cb-8d53-4b37-835b-591edcba96f1" path="/var/lib/kubelet/pods/7eb091cb-8d53-4b37-835b-591edcba96f1/volumes" Oct 29 23:30:27.488053 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:27.488035 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cecae6fc-a464-491b-945e-7344fca3eacd" path="/var/lib/kubelet/pods/cecae6fc-a464-491b-945e-7344fca3eacd/volumes" Oct 29 23:30:27.488498 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:27.488483 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2f3e7bb-6ca1-46a8-a146-cb7c5690f377" path="/var/lib/kubelet/pods/e2f3e7bb-6ca1-46a8-a146-cb7c5690f377/volumes" Oct 29 23:30:27.489197 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:27.489170 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea3ec3f-906d-4185-b80b-00c0f50bee8e" path="/var/lib/kubelet/pods/eea3ec3f-906d-4185-b80b-00c0f50bee8e/volumes" Oct 29 23:30:32.621245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.621211 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf"] Oct 29 23:30:32.622164 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.622146 5057 cpu_manager.go:414] "containerMap: removing container" podUID="595f0d6b-00c5-4a95-8881-8191e7de38ff" containerName="cleanup" Oct 29 23:30:32.622164 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.622163 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="595f0d6b-00c5-4a95-8881-8191e7de38ff" containerName="cleanup" Oct 29 23:30:32.622372 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.622178 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a2de74b6-1d73-4ab4-adaa-e081fbc0240d" containerName="cleanup" Oct 29 23:30:32.624276 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624244 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2de74b6-1d73-4ab4-adaa-e081fbc0240d" containerName="cleanup" Oct 29 23:30:32.624411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624286 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8" containerName="cleanup" Oct 29 23:30:32.624411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624296 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8" containerName="cleanup" Oct 29 23:30:32.624411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624306 5057 cpu_manager.go:414] "containerMap: removing container" podUID="cecae6fc-a464-491b-945e-7344fca3eacd" containerName="cleanup" Oct 29 23:30:32.624411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624313 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cecae6fc-a464-491b-945e-7344fca3eacd" containerName="cleanup" Oct 29 23:30:32.624411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624342 5057 cpu_manager.go:414] "containerMap: removing container" podUID="04c2e336-3da0-4816-8313-e8fdeb0edb5f" containerName="cleanup" Oct 29 23:30:32.624411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624349 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c2e336-3da0-4816-8313-e8fdeb0edb5f" containerName="cleanup" Oct 29 23:30:32.624411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624365 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7eb091cb-8d53-4b37-835b-591edcba96f1" containerName="cleanup" Oct 29 23:30:32.624411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624372 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eb091cb-8d53-4b37-835b-591edcba96f1" containerName="cleanup" Oct 29 23:30:32.624411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624387 5057 cpu_manager.go:414] "containerMap: removing container" podUID="13000cb3-436f-4fec-bb70-7e09a449dec0" containerName="cleanup" Oct 29 23:30:32.624411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624394 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13000cb3-436f-4fec-bb70-7e09a449dec0" containerName="cleanup" Oct 29 23:30:32.624411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624407 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7eb091cb-8d53-4b37-835b-591edcba96f1" containerName="cleanup" Oct 29 23:30:32.624411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624414 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eb091cb-8d53-4b37-835b-591edcba96f1" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624427 5057 cpu_manager.go:414] "containerMap: removing container" podUID="664c869e-4b17-41bb-946b-d2ead80eff08" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624436 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="664c869e-4b17-41bb-946b-d2ead80eff08" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624451 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e2f3e7bb-6ca1-46a8-a146-cb7c5690f377" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624458 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f3e7bb-6ca1-46a8-a146-cb7c5690f377" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624507 5057 cpu_manager.go:414] "containerMap: removing container" podUID="664c869e-4b17-41bb-946b-d2ead80eff08" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624514 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="664c869e-4b17-41bb-946b-d2ead80eff08" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624526 5057 cpu_manager.go:414] "containerMap: removing container" podUID="8e2a04ca-f403-49e9-bd0c-f9f9394e188b" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624533 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2a04ca-f403-49e9-bd0c-f9f9394e188b" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624545 5057 cpu_manager.go:414] "containerMap: removing container" podUID="8e2a04ca-f403-49e9-bd0c-f9f9394e188b" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624553 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2a04ca-f403-49e9-bd0c-f9f9394e188b" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624567 5057 cpu_manager.go:414] "containerMap: removing container" podUID="eea3ec3f-906d-4185-b80b-00c0f50bee8e" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624574 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea3ec3f-906d-4185-b80b-00c0f50bee8e" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624590 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7eb091cb-8d53-4b37-835b-591edcba96f1" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624597 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eb091cb-8d53-4b37-835b-591edcba96f1" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624613 5057 cpu_manager.go:414] "containerMap: removing container" podUID="13000cb3-436f-4fec-bb70-7e09a449dec0" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624620 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13000cb3-436f-4fec-bb70-7e09a449dec0" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624647 5057 cpu_manager.go:414] "containerMap: removing container" podUID="eea3ec3f-906d-4185-b80b-00c0f50bee8e" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624654 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea3ec3f-906d-4185-b80b-00c0f50bee8e" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624667 5057 cpu_manager.go:414] "containerMap: removing container" podUID="8e2a04ca-f403-49e9-bd0c-f9f9394e188b" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624674 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2a04ca-f403-49e9-bd0c-f9f9394e188b" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624686 5057 cpu_manager.go:414] "containerMap: removing container" podUID="04c2e336-3da0-4816-8313-e8fdeb0edb5f" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624694 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c2e336-3da0-4816-8313-e8fdeb0edb5f" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624705 5057 cpu_manager.go:414] "containerMap: removing container" podUID="cecae6fc-a464-491b-945e-7344fca3eacd" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624712 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cecae6fc-a464-491b-945e-7344fca3eacd" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624736 5057 cpu_manager.go:414] "containerMap: removing container" podUID="77571d53-cd5a-4b79-922b-3884bf5f7fcb" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624743 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77571d53-cd5a-4b79-922b-3884bf5f7fcb" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624756 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1c2a4a82-5ab9-4c15-8a1d-54da076ff3db" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624763 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2a4a82-5ab9-4c15-8a1d-54da076ff3db" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624779 5057 cpu_manager.go:414] "containerMap: removing container" podUID="77571d53-cd5a-4b79-922b-3884bf5f7fcb" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624786 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77571d53-cd5a-4b79-922b-3884bf5f7fcb" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624813 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a2de74b6-1d73-4ab4-adaa-e081fbc0240d" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624820 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2de74b6-1d73-4ab4-adaa-e081fbc0240d" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624833 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1c2a4a82-5ab9-4c15-8a1d-54da076ff3db" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624839 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2a4a82-5ab9-4c15-8a1d-54da076ff3db" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624853 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624860 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624873 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1c2a4a82-5ab9-4c15-8a1d-54da076ff3db" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624882 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2a4a82-5ab9-4c15-8a1d-54da076ff3db" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624900 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a2de74b6-1d73-4ab4-adaa-e081fbc0240d" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624908 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2de74b6-1d73-4ab4-adaa-e081fbc0240d" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624926 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e2f3e7bb-6ca1-46a8-a146-cb7c5690f377" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624933 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f3e7bb-6ca1-46a8-a146-cb7c5690f377" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624953 5057 cpu_manager.go:414] "containerMap: removing container" podUID="664c869e-4b17-41bb-946b-d2ead80eff08" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624960 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="664c869e-4b17-41bb-946b-d2ead80eff08" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624971 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624978 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624988 5057 cpu_manager.go:414] "containerMap: removing container" podUID="595f0d6b-00c5-4a95-8881-8191e7de38ff" containerName="cleanup" Oct 29 23:30:32.625070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.624994 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="595f0d6b-00c5-4a95-8881-8191e7de38ff" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625274 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c2a4a82-5ab9-4c15-8a1d-54da076ff3db" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625292 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="cecae6fc-a464-491b-945e-7344fca3eacd" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625302 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="77571d53-cd5a-4b79-922b-3884bf5f7fcb" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625315 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2de74b6-1d73-4ab4-adaa-e081fbc0240d" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625323 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c2a4a82-5ab9-4c15-8a1d-54da076ff3db" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625336 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea3ec3f-906d-4185-b80b-00c0f50bee8e" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625345 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625353 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c2a4a82-5ab9-4c15-8a1d-54da076ff3db" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625363 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2de74b6-1d73-4ab4-adaa-e081fbc0240d" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625374 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f3e7bb-6ca1-46a8-a146-cb7c5690f377" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625382 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="77571d53-cd5a-4b79-922b-3884bf5f7fcb" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625392 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="664c869e-4b17-41bb-946b-d2ead80eff08" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625400 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="595f0d6b-00c5-4a95-8881-8191e7de38ff" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625410 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2de74b6-1d73-4ab4-adaa-e081fbc0240d" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625418 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625426 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="cecae6fc-a464-491b-945e-7344fca3eacd" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625437 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="04c2e336-3da0-4816-8313-e8fdeb0edb5f" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625445 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eb091cb-8d53-4b37-835b-591edcba96f1" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625454 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="13000cb3-436f-4fec-bb70-7e09a449dec0" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625466 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f3e7bb-6ca1-46a8-a146-cb7c5690f377" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625474 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="595f0d6b-00c5-4a95-8881-8191e7de38ff" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625483 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="664c869e-4b17-41bb-946b-d2ead80eff08" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625491 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2a04ca-f403-49e9-bd0c-f9f9394e188b" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625501 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2a04ca-f403-49e9-bd0c-f9f9394e188b" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625509 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eb091cb-8d53-4b37-835b-591edcba96f1" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625520 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="13000cb3-436f-4fec-bb70-7e09a449dec0" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625531 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="13000cb3-436f-4fec-bb70-7e09a449dec0" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625543 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="04c2e336-3da0-4816-8313-e8fdeb0edb5f" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625554 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea3ec3f-906d-4185-b80b-00c0f50bee8e" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625563 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="04c2e336-3da0-4816-8313-e8fdeb0edb5f" containerName="cleanup" Oct 29 23:30:32.627920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.625574 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="cecae6fc-a464-491b-945e-7344fca3eacd" containerName="cleanup" Oct 29 23:30:32.632620 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.632598 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf" Oct 29 23:30:32.653432 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.653406 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf"] Oct 29 23:30:32.674657 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.674617 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-90f683\"" Oct 29 23:30:32.674856 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.674820 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-90f683-dockercfg-hqfb5\"" Oct 29 23:30:32.811260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.806077 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljzkd\" (UniqueName: \"kubernetes.io/projected/483d8088-c440-44da-ab9e-b7f4edd88410-kube-api-access-ljzkd\") pod \"payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf\" (UID: \"483d8088-c440-44da-ab9e-b7f4edd88410\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf" Oct 29 23:30:32.811260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:32.806288 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/483d8088-c440-44da-ab9e-b7f4edd88410-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf\" (UID: \"483d8088-c440-44da-ab9e-b7f4edd88410\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf" Oct 29 23:30:33.032210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:33.021178 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljzkd\" (UniqueName: \"kubernetes.io/projected/483d8088-c440-44da-ab9e-b7f4edd88410-kube-api-access-ljzkd\") pod \"payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf\" (UID: \"483d8088-c440-44da-ab9e-b7f4edd88410\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf" Oct 29 23:30:33.032210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:33.021397 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/483d8088-c440-44da-ab9e-b7f4edd88410-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf\" (UID: \"483d8088-c440-44da-ab9e-b7f4edd88410\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf" Oct 29 23:30:33.032210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:33.022623 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/483d8088-c440-44da-ab9e-b7f4edd88410-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf\" (UID: \"483d8088-c440-44da-ab9e-b7f4edd88410\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf" Oct 29 23:30:33.147009 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:33.146932 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljzkd\" (UniqueName: \"kubernetes.io/projected/483d8088-c440-44da-ab9e-b7f4edd88410-kube-api-access-ljzkd\") pod \"payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf\" (UID: \"483d8088-c440-44da-ab9e-b7f4edd88410\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf" Oct 29 23:30:33.265812 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:33.265773 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf" Oct 29 23:30:33.581038 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:33.580975 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf"] Oct 29 23:30:33.581676 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:30:33.581646 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483d8088_c440_44da_ab9e_b7f4edd88410.slice/crio-908278476d6f99d9d1ce5592d51ac2914aadbbc97986b2c41d6e32c35beaa137 WatchSource:0}: Error finding container 908278476d6f99d9d1ce5592d51ac2914aadbbc97986b2c41d6e32c35beaa137: Status 404 returned error can't find the container with id 908278476d6f99d9d1ce5592d51ac2914aadbbc97986b2c41d6e32c35beaa137 Oct 29 23:30:33.915577 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:30:33.915495 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:30:34.541928 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:34.541882 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf" event={"ID":"483d8088-c440-44da-ab9e-b7f4edd88410","Type":"ContainerStarted","Data":"7af80cd6829641f615214cb1924bfe8e2dcd7846098a0bea85a7c521a682a069"} Oct 29 23:30:34.542151 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:34.542014 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf" event={"ID":"483d8088-c440-44da-ab9e-b7f4edd88410","Type":"ContainerStarted","Data":"908278476d6f99d9d1ce5592d51ac2914aadbbc97986b2c41d6e32c35beaa137"} Oct 29 23:30:34.542328 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:34.542305 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf" Oct 29 23:30:34.590415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:34.590360 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf" podStartSLOduration=2.590347403 podStartE2EDuration="2.590347403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:34.585836718 +0000 UTC m=+2881.237335248" watchObservedRunningTime="2026-10-29 23:30:34.590347403 +0000 UTC m=+2881.241845932" Oct 29 23:30:38.270287 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:38.270254 5057 scope.go:122] "RemoveContainer" containerID="ee2eff3bf2dfb04550ad282099e5e96ac9c94b479a0a740c1e2b53fa35229b6a" Oct 29 23:30:45.557866 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:45.557839 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf" Oct 29 23:30:50.035238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:50.035172 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:30:52.135211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:52.134966 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-58ks4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:30:52.135211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:30:52.135051 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" podUID="44918f77-9e08-4a3e-ba96-95763437e8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:31:00.798711 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.798660 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr"] Oct 29 23:31:00.800993 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.800963 5057 cpu_manager.go:414] "containerMap: removing container" podUID="cecae6fc-a464-491b-945e-7344fca3eacd" containerName="cleanup" Oct 29 23:31:00.801212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.801181 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cecae6fc-a464-491b-945e-7344fca3eacd" containerName="cleanup" Oct 29 23:31:00.801369 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.801359 5057 cpu_manager.go:414] "containerMap: removing container" podUID="13000cb3-436f-4fec-bb70-7e09a449dec0" containerName="cleanup" Oct 29 23:31:00.801459 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.801448 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13000cb3-436f-4fec-bb70-7e09a449dec0" containerName="cleanup" Oct 29 23:31:00.801553 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.801544 5057 cpu_manager.go:414] "containerMap: removing container" podUID="eea3ec3f-906d-4185-b80b-00c0f50bee8e" containerName="cleanup" Oct 29 23:31:00.801628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.801619 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea3ec3f-906d-4185-b80b-00c0f50bee8e" containerName="cleanup" Oct 29 23:31:00.801716 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.801707 5057 cpu_manager.go:414] "containerMap: removing container" podUID="04c2e336-3da0-4816-8313-e8fdeb0edb5f" containerName="cleanup" Oct 29 23:31:00.801798 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.801788 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c2e336-3da0-4816-8313-e8fdeb0edb5f" containerName="cleanup" Oct 29 23:31:00.801909 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.801879 5057 cpu_manager.go:414] "containerMap: removing container" podUID="77571d53-cd5a-4b79-922b-3884bf5f7fcb" containerName="cleanup" Oct 29 23:31:00.801996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.801984 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77571d53-cd5a-4b79-922b-3884bf5f7fcb" containerName="cleanup" Oct 29 23:31:00.802100 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.802092 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e2f3e7bb-6ca1-46a8-a146-cb7c5690f377" containerName="cleanup" Oct 29 23:31:00.802178 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.802169 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f3e7bb-6ca1-46a8-a146-cb7c5690f377" containerName="cleanup" Oct 29 23:31:00.802290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.802279 5057 cpu_manager.go:414] "containerMap: removing container" podUID="595f0d6b-00c5-4a95-8881-8191e7de38ff" containerName="cleanup" Oct 29 23:31:00.802371 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.802362 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="595f0d6b-00c5-4a95-8881-8191e7de38ff" containerName="cleanup" Oct 29 23:31:00.803105 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.803083 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea3ec3f-906d-4185-b80b-00c0f50bee8e" containerName="cleanup" Oct 29 23:31:00.803263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.803248 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="77571d53-cd5a-4b79-922b-3884bf5f7fcb" containerName="cleanup" Oct 29 23:31:00.803360 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.803350 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="664c869e-4b17-41bb-946b-d2ead80eff08" containerName="cleanup" Oct 29 23:31:00.803457 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.803448 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e01bb8e-5fd2-4f7c-acd3-aa87827f3cb8" containerName="cleanup" Oct 29 23:31:00.803541 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.803532 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f3e7bb-6ca1-46a8-a146-cb7c5690f377" containerName="cleanup" Oct 29 23:31:00.803621 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.803613 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="595f0d6b-00c5-4a95-8881-8191e7de38ff" containerName="cleanup" Oct 29 23:31:00.803701 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.803693 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2a04ca-f403-49e9-bd0c-f9f9394e188b" containerName="cleanup" Oct 29 23:31:00.803787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.803777 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eb091cb-8d53-4b37-835b-591edcba96f1" containerName="cleanup" Oct 29 23:31:00.810858 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.810832 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" Oct 29 23:31:00.875211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.862177 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-45036b\"/\"e2e-cmp-legacy-fe3cd6-provider-cred\"" Oct 29 23:31:00.875211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.862919 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-45036b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-45036b\"" Oct 29 23:31:00.875211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.863010 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-45036b\"/\"openshift-service-ca.crt\"" Oct 29 23:31:00.875211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.863066 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-45036b\"/\"routing-overlay\"" Oct 29 23:31:00.875211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.863144 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-45036b\"/\"kube-root-ca.crt\"" Oct 29 23:31:00.875211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.863256 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-45036b\"/\"payload-processing-external-model-e2e--86ce1f4b-dockercfg-rrrhr\"" Oct 29 23:31:00.884347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.883575 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ae6273c3-6af4-40fe-b208-8310228fc9a9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-6986297vr\" (UID: \"ae6273c3-6af4-40fe-b208-8310228fc9a9\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" Oct 29 23:31:00.884347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.883868 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wzfb\" (UniqueName: \"kubernetes.io/projected/ae6273c3-6af4-40fe-b208-8310228fc9a9-kube-api-access-9wzfb\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-6986297vr\" (UID: \"ae6273c3-6af4-40fe-b208-8310228fc9a9\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" Oct 29 23:31:00.884347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.883931 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae6273c3-6af4-40fe-b208-8310228fc9a9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-6986297vr\" (UID: \"ae6273c3-6af4-40fe-b208-8310228fc9a9\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" Oct 29 23:31:00.884347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.884097 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ae6273c3-6af4-40fe-b208-8310228fc9a9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-6986297vr\" (UID: \"ae6273c3-6af4-40fe-b208-8310228fc9a9\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" Oct 29 23:31:01.002795 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.985978 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wzfb\" (UniqueName: \"kubernetes.io/projected/ae6273c3-6af4-40fe-b208-8310228fc9a9-kube-api-access-9wzfb\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-6986297vr\" (UID: \"ae6273c3-6af4-40fe-b208-8310228fc9a9\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" Oct 29 23:31:01.002795 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.986061 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae6273c3-6af4-40fe-b208-8310228fc9a9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-6986297vr\" (UID: \"ae6273c3-6af4-40fe-b208-8310228fc9a9\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" Oct 29 23:31:01.002795 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.986211 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ae6273c3-6af4-40fe-b208-8310228fc9a9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-6986297vr\" (UID: \"ae6273c3-6af4-40fe-b208-8310228fc9a9\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" Oct 29 23:31:01.002795 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.986340 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ae6273c3-6af4-40fe-b208-8310228fc9a9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-6986297vr\" (UID: \"ae6273c3-6af4-40fe-b208-8310228fc9a9\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" Oct 29 23:31:01.002795 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:00.987353 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ae6273c3-6af4-40fe-b208-8310228fc9a9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-6986297vr\" (UID: \"ae6273c3-6af4-40fe-b208-8310228fc9a9\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" Oct 29 23:31:01.050270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:01.025586 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae6273c3-6af4-40fe-b208-8310228fc9a9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-6986297vr\" (UID: \"ae6273c3-6af4-40fe-b208-8310228fc9a9\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" Oct 29 23:31:01.212209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:01.155478 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ae6273c3-6af4-40fe-b208-8310228fc9a9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-6986297vr\" (UID: \"ae6273c3-6af4-40fe-b208-8310228fc9a9\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" Oct 29 23:31:01.212209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:01.185629 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr"] Oct 29 23:31:01.320539 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:01.320449 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wzfb\" (UniqueName: \"kubernetes.io/projected/ae6273c3-6af4-40fe-b208-8310228fc9a9-kube-api-access-9wzfb\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-6986297vr\" (UID: \"ae6273c3-6af4-40fe-b208-8310228fc9a9\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" Oct 29 23:31:01.424117 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:01.424072 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" Oct 29 23:31:01.777883 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:31:01.777847 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6273c3_6af4_40fe_b208_8310228fc9a9.slice/crio-7ea1fed1639aaa29908dcc8326cef7c38f917e867ed6eea27704049bb669e86b WatchSource:0}: Error finding container 7ea1fed1639aaa29908dcc8326cef7c38f917e867ed6eea27704049bb669e86b: Status 404 returned error can't find the container with id 7ea1fed1639aaa29908dcc8326cef7c38f917e867ed6eea27704049bb669e86b Oct 29 23:31:01.778116 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:01.778094 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr"] Oct 29 23:31:02.344497 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:02.344454 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" event={"ID":"ae6273c3-6af4-40fe-b208-8310228fc9a9","Type":"ContainerStarted","Data":"5e79d78025b87ba0705ac9e20e246c730da274db82c7e912d3acdc2e9fefcd3d"} Oct 29 23:31:02.344962 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:02.344530 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" Oct 29 23:31:02.344962 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:02.344554 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" event={"ID":"ae6273c3-6af4-40fe-b208-8310228fc9a9","Type":"ContainerStarted","Data":"7ea1fed1639aaa29908dcc8326cef7c38f917e867ed6eea27704049bb669e86b"} Oct 29 23:31:02.391840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:02.391786 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" podStartSLOduration=2.391771693 podStartE2EDuration="2.391771693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:02.390264252 +0000 UTC m=+2909.041762781" watchObservedRunningTime="2026-10-29 23:31:02.391771693 +0000 UTC m=+2909.043270222" Oct 29 23:31:03.915858 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:31:03.915820 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:31:13.357586 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:13.357542 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" Oct 29 23:31:17.551808 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:17.551726 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq"] Oct 29 23:31:17.557775 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:17.557751 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:17.576005 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:17.575954 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5cd0f2-openshift-default-dockercfg-q2mg6\"" Oct 29 23:31:17.598895 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:17.598862 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq"] Oct 29 23:31:17.635615 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:17.635579 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8645590d-04a6-4cff-a65e-281d5cb7f059-istiod-ca-cert\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:17.635811 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:17.635639 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-data\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:17.635811 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:17.635673 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-podinfo\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:17.635811 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:17.635741 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-credential-socket\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:17.635811 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:17.635770 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-workload-certs\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:17.636047 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:17.635912 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-token\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:17.636047 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:17.635938 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-envoy\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:17.636047 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:17.635999 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98hfl\" (UniqueName: \"kubernetes.io/projected/8645590d-04a6-4cff-a65e-281d5cb7f059-kube-api-access-98hfl\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:17.636223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:17.636168 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-workload-socket\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:18.078231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.071894 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98hfl\" (UniqueName: \"kubernetes.io/projected/8645590d-04a6-4cff-a65e-281d5cb7f059-kube-api-access-98hfl\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:18.078231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.072128 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-workload-socket\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:18.078231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.072323 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8645590d-04a6-4cff-a65e-281d5cb7f059-istiod-ca-cert\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:18.078231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.072372 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-data\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:18.078231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.072402 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-podinfo\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:18.078231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.072480 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-credential-socket\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:18.078231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.072510 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-workload-certs\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:18.078231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.072689 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-token\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:18.121810 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.083232 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8645590d-04a6-4cff-a65e-281d5cb7f059-istiod-ca-cert\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:18.121810 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.083559 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-workload-socket\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:18.121810 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.083799 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-workload-certs\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:18.121810 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.084023 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-credential-socket\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:18.121810 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.084286 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-data\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:18.238286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.072720 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-envoy\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:18.239171 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.239096 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98hfl\" (UniqueName: \"kubernetes.io/projected/8645590d-04a6-4cff-a65e-281d5cb7f059-kube-api-access-98hfl\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:18.269237 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.268641 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-podinfo\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:18.269237 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.268923 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-envoy\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:18.269237 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.269156 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-token\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:18.493035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.492999 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:18.826880 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:31:18.826850 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8645590d_04a6_4cff_a65e_281d5cb7f059.slice/crio-d57ec618f6cdec01c3e0fb8e498c0bf0c9b2fd124439504036e90c11daa4ebc0 WatchSource:0}: Error finding container d57ec618f6cdec01c3e0fb8e498c0bf0c9b2fd124439504036e90c11daa4ebc0: Status 404 returned error can't find the container with id d57ec618f6cdec01c3e0fb8e498c0bf0c9b2fd124439504036e90c11daa4ebc0 Oct 29 23:31:18.830280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.830254 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq"] Oct 29 23:31:18.830912 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.830868 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:31:18.830975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.830957 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:31:18.831012 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:18.831001 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:31:19.565761 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:19.565711 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" event={"ID":"8645590d-04a6-4cff-a65e-281d5cb7f059","Type":"ContainerStarted","Data":"4c3c6e40a4cae75758e4b48e4f060d61288fe485a7a8bb2d757735edf820d597"} Oct 29 23:31:19.565959 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:19.565783 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" event={"ID":"8645590d-04a6-4cff-a65e-281d5cb7f059","Type":"ContainerStarted","Data":"d57ec618f6cdec01c3e0fb8e498c0bf0c9b2fd124439504036e90c11daa4ebc0"} Oct 29 23:31:19.616647 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:19.616588 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" podStartSLOduration=2.616574735 podStartE2EDuration="2.616574735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:19.61502876 +0000 UTC m=+2926.266527292" watchObservedRunningTime="2026-10-29 23:31:19.616574735 +0000 UTC m=+2926.268073264" Oct 29 23:31:20.494125 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:20.494083 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:20.502221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:20.502173 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:20.575511 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:20.575477 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:20.576418 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:20.576394 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:31:23.920105 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:23.920066 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d"] Oct 29 23:31:23.960213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:23.951714 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d" Oct 29 23:31:24.017210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:24.016655 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5cd0f2\"" Oct 29 23:31:24.017210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:24.016855 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5cd0f2-dockercfg-frt4g\"" Oct 29 23:31:24.395995 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:24.395476 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0f270e1-9bac-4a4c-a472-4d4b07fa7894-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d\" (UID: \"d0f270e1-9bac-4a4c-a472-4d4b07fa7894\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d" Oct 29 23:31:24.395995 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:24.395659 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzjth\" (UniqueName: \"kubernetes.io/projected/d0f270e1-9bac-4a4c-a472-4d4b07fa7894-kube-api-access-kzjth\") pod \"payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d\" (UID: \"d0f270e1-9bac-4a4c-a472-4d4b07fa7894\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d" Oct 29 23:31:24.420960 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:24.420931 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d"] Oct 29 23:31:24.497200 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:24.497152 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0f270e1-9bac-4a4c-a472-4d4b07fa7894-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d\" (UID: \"d0f270e1-9bac-4a4c-a472-4d4b07fa7894\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d" Oct 29 23:31:24.497412 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:24.497350 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzjth\" (UniqueName: \"kubernetes.io/projected/d0f270e1-9bac-4a4c-a472-4d4b07fa7894-kube-api-access-kzjth\") pod \"payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d\" (UID: \"d0f270e1-9bac-4a4c-a472-4d4b07fa7894\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d" Oct 29 23:31:24.497996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:24.497976 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0f270e1-9bac-4a4c-a472-4d4b07fa7894-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d\" (UID: \"d0f270e1-9bac-4a4c-a472-4d4b07fa7894\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d" Oct 29 23:31:24.523143 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:24.523074 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzjth\" (UniqueName: \"kubernetes.io/projected/d0f270e1-9bac-4a4c-a472-4d4b07fa7894-kube-api-access-kzjth\") pod \"payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d\" (UID: \"d0f270e1-9bac-4a4c-a472-4d4b07fa7894\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d" Oct 29 23:31:24.758465 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:24.758421 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d" Oct 29 23:31:25.068852 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:31:25.068819 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f270e1_9bac_4a4c_a472_4d4b07fa7894.slice/crio-4d6a75212032b0728aa782edc2ab5890aeddb74811504bd8ba36bc6631d7f48a WatchSource:0}: Error finding container 4d6a75212032b0728aa782edc2ab5890aeddb74811504bd8ba36bc6631d7f48a: Status 404 returned error can't find the container with id 4d6a75212032b0728aa782edc2ab5890aeddb74811504bd8ba36bc6631d7f48a Oct 29 23:31:25.069645 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:25.069624 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d"] Oct 29 23:31:25.624760 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:25.624725 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d" event={"ID":"d0f270e1-9bac-4a4c-a472-4d4b07fa7894","Type":"ContainerStarted","Data":"7154e59da8308cb34079990640813e5b9ea588618ff78f26e51467078dfd4e56"} Oct 29 23:31:25.624960 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:25.624789 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d" event={"ID":"d0f270e1-9bac-4a4c-a472-4d4b07fa7894","Type":"ContainerStarted","Data":"4d6a75212032b0728aa782edc2ab5890aeddb74811504bd8ba36bc6631d7f48a"} Oct 29 23:31:25.624960 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:25.624820 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d" Oct 29 23:31:25.673684 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:25.673626 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d" podStartSLOduration=2.6736119929999997 podStartE2EDuration="2.673611993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:25.668911477 +0000 UTC m=+2932.320410011" watchObservedRunningTime="2026-10-29 23:31:25.673611993 +0000 UTC m=+2932.325110568" Oct 29 23:31:33.914517 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:31:33.914478 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:31:36.658744 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:36.658718 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d" Oct 29 23:31:44.078519 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:44.078476 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:31:46.630908 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:46.630851 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:31:46.832212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:46.784560 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:31:47.025210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:47.022277 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:31:47.037321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:47.037273 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:31:49.533249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:31:49.511084 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 29 23:32:03.251503 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.251462 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s"] Oct 29 23:32:03.259849 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.259822 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.279114 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.279070 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-51462d-openshift-default-dockercfg-rt7w7\"" Oct 29 23:32:03.301355 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.301311 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s"] Oct 29 23:32:03.302604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.302577 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-workload-socket\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.302775 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.302748 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1729a889-e3d1-43df-89d1-f3356bc34a46-istiod-ca-cert\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.302855 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.302795 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-podinfo\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.303007 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.302990 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-envoy\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.303301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.303272 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-workload-certs\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.303433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.303318 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-credential-socket\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.303433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.303387 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-data\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.303597 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.303575 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht76t\" (UniqueName: \"kubernetes.io/projected/1729a889-e3d1-43df-89d1-f3356bc34a46-kube-api-access-ht76t\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.303637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.303616 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-token\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.408314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.408157 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1729a889-e3d1-43df-89d1-f3356bc34a46-istiod-ca-cert\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.456176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.420736 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-podinfo\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.456828 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.456803 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-envoy\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.457339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.457316 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-workload-certs\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.457516 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.457497 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-credential-socket\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.457669 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.457654 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-data\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.457963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.457946 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht76t\" (UniqueName: \"kubernetes.io/projected/1729a889-e3d1-43df-89d1-f3356bc34a46-kube-api-access-ht76t\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.458107 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.458090 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-token\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.458286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.458263 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-workload-socket\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.504263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.469814 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-credential-socket\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.542372 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.512917 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1729a889-e3d1-43df-89d1-f3356bc34a46-istiod-ca-cert\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.556256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.555578 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-data\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.556256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.556158 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-workload-socket\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.568632 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.561260 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-workload-certs\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.766522 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.765413 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-envoy\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.767290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.767257 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-podinfo\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.783896 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.783825 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht76t\" (UniqueName: \"kubernetes.io/projected/1729a889-e3d1-43df-89d1-f3356bc34a46-kube-api-access-ht76t\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.784634 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.784606 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-token\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.922322 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:03.921055 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:03.983266 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:32:03.983143 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:32:04.295361 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:04.295328 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s"] Oct 29 23:32:04.296820 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:32:04.296790 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1729a889_e3d1_43df_89d1_f3356bc34a46.slice/crio-a33bbc4fcf510252f811f4e65235b1d510f4bf19b955187d82b211f861287429 WatchSource:0}: Error finding container a33bbc4fcf510252f811f4e65235b1d510f4bf19b955187d82b211f861287429: Status 404 returned error can't find the container with id a33bbc4fcf510252f811f4e65235b1d510f4bf19b955187d82b211f861287429 Oct 29 23:32:04.300216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:04.300147 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:32:04.300304 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:04.300237 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:32:04.300304 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:04.300268 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:32:04.574482 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:04.574382 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" event={"ID":"1729a889-e3d1-43df-89d1-f3356bc34a46","Type":"ContainerStarted","Data":"233a3bc3f3ec7b97f979dc0789e6e4b2cf6c87a2b201fcec7d6967f4e0bc98c5"} Oct 29 23:32:04.574482 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:04.574439 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" event={"ID":"1729a889-e3d1-43df-89d1-f3356bc34a46","Type":"ContainerStarted","Data":"a33bbc4fcf510252f811f4e65235b1d510f4bf19b955187d82b211f861287429"} Oct 29 23:32:04.628732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:04.628668 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" podStartSLOduration=1.628647212 podStartE2EDuration="1.628647212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:04.622697685 +0000 UTC m=+2971.274196217" watchObservedRunningTime="2026-10-29 23:32:04.628647212 +0000 UTC m=+2971.280145747" Oct 29 23:32:04.926144 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:04.926114 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:05.935508 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:05.935472 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:06.592323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:06.592298 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:06.593308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:06.593285 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:32:33.914955 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:32:33.914919 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:32:35.112033 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:35.111997 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:32:35.127865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:32:35.127839 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:33:03.915830 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:33:03.915785 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:33:06.209918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:06.209883 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w"] Oct 29 23:33:06.215176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:06.215150 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w" Oct 29 23:33:06.235905 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:06.235873 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w"] Oct 29 23:33:06.315714 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:06.301650 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9965bf-dockercfg-j9llb\"" Oct 29 23:33:06.380318 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:06.372435 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9965bf\"" Oct 29 23:33:06.631683 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:06.430780 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f2943d7-12ce-4123-a75b-bbbd49a823a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w\" (UID: \"2f2943d7-12ce-4123-a75b-bbbd49a823a5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w" Oct 29 23:33:06.632264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:06.632231 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4srhw\" (UniqueName: \"kubernetes.io/projected/2f2943d7-12ce-4123-a75b-bbbd49a823a5-kube-api-access-4srhw\") pod \"payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w\" (UID: \"2f2943d7-12ce-4123-a75b-bbbd49a823a5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w" Oct 29 23:33:06.779639 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:06.779593 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f2943d7-12ce-4123-a75b-bbbd49a823a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w\" (UID: \"2f2943d7-12ce-4123-a75b-bbbd49a823a5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w" Oct 29 23:33:06.779904 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:06.779888 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4srhw\" (UniqueName: \"kubernetes.io/projected/2f2943d7-12ce-4123-a75b-bbbd49a823a5-kube-api-access-4srhw\") pod \"payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w\" (UID: \"2f2943d7-12ce-4123-a75b-bbbd49a823a5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w" Oct 29 23:33:06.795229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:06.784631 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f2943d7-12ce-4123-a75b-bbbd49a823a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w\" (UID: \"2f2943d7-12ce-4123-a75b-bbbd49a823a5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w" Oct 29 23:33:06.808448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:06.808417 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4srhw\" (UniqueName: \"kubernetes.io/projected/2f2943d7-12ce-4123-a75b-bbbd49a823a5-kube-api-access-4srhw\") pod \"payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w\" (UID: \"2f2943d7-12ce-4123-a75b-bbbd49a823a5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w" Oct 29 23:33:06.831573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:06.831330 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w" Oct 29 23:33:07.176253 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:33:07.176221 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2943d7_12ce_4123_a75b_bbbd49a823a5.slice/crio-ced5298298089416bc0beae8f89336b5625f28e54d7f596f5fc738a00c153ee0 WatchSource:0}: Error finding container ced5298298089416bc0beae8f89336b5625f28e54d7f596f5fc738a00c153ee0: Status 404 returned error can't find the container with id ced5298298089416bc0beae8f89336b5625f28e54d7f596f5fc738a00c153ee0 Oct 29 23:33:07.176632 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:07.176603 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w"] Oct 29 23:33:08.094752 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:08.094718 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w" event={"ID":"2f2943d7-12ce-4123-a75b-bbbd49a823a5","Type":"ContainerStarted","Data":"fade27f70677cd84512a6f4825d53d321b69cde4464454db94007295b98b3fb4"} Oct 29 23:33:08.095222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:08.094779 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w" event={"ID":"2f2943d7-12ce-4123-a75b-bbbd49a823a5","Type":"ContainerStarted","Data":"ced5298298089416bc0beae8f89336b5625f28e54d7f596f5fc738a00c153ee0"} Oct 29 23:33:08.095222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:08.094808 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w" Oct 29 23:33:08.144962 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:08.144895 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w" podStartSLOduration=2.14487645 podStartE2EDuration="2.14487645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:08.143587972 +0000 UTC m=+3034.795086503" watchObservedRunningTime="2026-10-29 23:33:08.14487645 +0000 UTC m=+3034.796374981" Oct 29 23:33:19.134977 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:19.134947 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w" Oct 29 23:33:22.715856 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:22.715822 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh"] Oct 29 23:33:22.716365 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:22.716041 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" podUID="5c287175-5866-4c13-b806-5839582839cc" containerName="istio-proxy" containerID="cri-o://fc0b40b5017859e1b0803ac831d27092d19986023b421294bf6e31b19f365e5d" gracePeriod=30 Oct 29 23:33:23.019309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:23.013921 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:33:26.401518 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:26.391634 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb"] Oct 29 23:33:26.402398 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:26.402356 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" podUID="c5db1599-6906-4dcd-8182-787026e2e745" containerName="istio-proxy" containerID="cri-o://baa166cb6c5d9234be7485ef6531651d68a362859ac6d517c99c59e258748111" gracePeriod=30 Oct 29 23:33:28.059902 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.059870 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:33:28.243471 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.243436 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c287175-5866-4c13-b806-5839582839cc-kube-api-access-xwf7x\" (UniqueName: \"kubernetes.io/projected/5c287175-5866-4c13-b806-5839582839cc-kube-api-access-xwf7x\") pod \"5c287175-5866-4c13-b806-5839582839cc\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " Oct 29 23:33:28.243715 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.243559 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-workload-certs\") pod \"5c287175-5866-4c13-b806-5839582839cc\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " Oct 29 23:33:28.243715 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.243641 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c287175-5866-4c13-b806-5839582839cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c287175-5866-4c13-b806-5839582839cc-istiod-ca-cert\") pod \"5c287175-5866-4c13-b806-5839582839cc\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " Oct 29 23:33:28.243864 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.243794 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c287175-5866-4c13-b806-5839582839cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c287175-5866-4c13-b806-5839582839cc-istio-podinfo\") pod \"5c287175-5866-4c13-b806-5839582839cc\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " Oct 29 23:33:28.243864 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.243829 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-workload-socket\") pod \"5c287175-5866-4c13-b806-5839582839cc\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " Oct 29 23:33:28.243995 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.243904 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-workload-certs" pod "5c287175-5866-4c13-b806-5839582839cc" (UID: "5c287175-5866-4c13-b806-5839582839cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:28.244161 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.244130 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-istio-envoy\") pod \"5c287175-5866-4c13-b806-5839582839cc\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " Oct 29 23:33:28.244287 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.244145 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-workload-socket" pod "5c287175-5866-4c13-b806-5839582839cc" (UID: "5c287175-5866-4c13-b806-5839582839cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:28.244287 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.244208 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c287175-5866-4c13-b806-5839582839cc-istio-token\" (UniqueName: \"kubernetes.io/projected/5c287175-5866-4c13-b806-5839582839cc-istio-token\") pod \"5c287175-5866-4c13-b806-5839582839cc\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " Oct 29 23:33:28.244287 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.244252 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-credential-socket\") pod \"5c287175-5866-4c13-b806-5839582839cc\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " Oct 29 23:33:28.244458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.244312 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-istio-data\") pod \"5c287175-5866-4c13-b806-5839582839cc\" (UID: \"5c287175-5866-4c13-b806-5839582839cc\") " Oct 29 23:33:28.244646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.244623 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c287175-5866-4c13-b806-5839582839cc-istiod-ca-cert" pod "5c287175-5866-4c13-b806-5839582839cc" (UID: "5c287175-5866-4c13-b806-5839582839cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:28.244832 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.244717 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-credential-socket" pod "5c287175-5866-4c13-b806-5839582839cc" (UID: "5c287175-5866-4c13-b806-5839582839cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:28.245002 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.244976 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-istio-data" pod "5c287175-5866-4c13-b806-5839582839cc" (UID: "5c287175-5866-4c13-b806-5839582839cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:28.245257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.245238 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:28.245334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.245259 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c287175-5866-4c13-b806-5839582839cc-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:28.245334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.245274 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:28.245334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.245287 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:28.245334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.245302 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:28.247635 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.247604 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-istio-envoy" pod "5c287175-5866-4c13-b806-5839582839cc" (UID: "5c287175-5866-4c13-b806-5839582839cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:28.248743 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.248710 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c287175-5866-4c13-b806-5839582839cc-istio-token" pod "5c287175-5866-4c13-b806-5839582839cc" (UID: "5c287175-5866-4c13-b806-5839582839cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:28.255310 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.255238 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c287175-5866-4c13-b806-5839582839cc-istio-podinfo" pod "5c287175-5866-4c13-b806-5839582839cc" (UID: "5c287175-5866-4c13-b806-5839582839cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:33:28.256643 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.256617 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c287175-5866-4c13-b806-5839582839cc-kube-api-access-xwf7x" pod "5c287175-5866-4c13-b806-5839582839cc" (UID: "5c287175-5866-4c13-b806-5839582839cc"). InnerVolumeSpecName "kube-api-access-xwf7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:28.411620 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.406031 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwf7x\" (UniqueName: \"kubernetes.io/projected/5c287175-5866-4c13-b806-5839582839cc-kube-api-access-xwf7x\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:28.411932 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.411906 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c287175-5866-4c13-b806-5839582839cc-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:28.412162 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.412110 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c287175-5866-4c13-b806-5839582839cc-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:28.412323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.412310 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c287175-5866-4c13-b806-5839582839cc-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:28.498245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.498160 5057 generic.go:356] "Generic (PLEG): container finished" podID="5c287175-5866-4c13-b806-5839582839cc" containerID="fc0b40b5017859e1b0803ac831d27092d19986023b421294bf6e31b19f365e5d" exitCode=0 Oct 29 23:33:28.498442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.498253 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" event={"ID":"5c287175-5866-4c13-b806-5839582839cc","Type":"ContainerDied","Data":"fc0b40b5017859e1b0803ac831d27092d19986023b421294bf6e31b19f365e5d"} Oct 29 23:33:28.498442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.498292 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" Oct 29 23:33:28.498442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.498328 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh" event={"ID":"5c287175-5866-4c13-b806-5839582839cc","Type":"ContainerDied","Data":"83e9057a3cbb91884a3e76fc04619aad6f8f6c8a33c6105bf682a57348f4a405"} Oct 29 23:33:28.498442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.498359 5057 scope.go:122] "RemoveContainer" containerID="fc0b40b5017859e1b0803ac831d27092d19986023b421294bf6e31b19f365e5d" Oct 29 23:33:28.513348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.513328 5057 scope.go:122] "RemoveContainer" containerID="081524ad795f0d79617a18403c6e36507788c2a4e9441e0832a1e3ccfbb6fbd0" Oct 29 23:33:28.527468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.527170 5057 scope.go:122] "RemoveContainer" containerID="fc0b40b5017859e1b0803ac831d27092d19986023b421294bf6e31b19f365e5d" Oct 29 23:33:28.527639 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:33:28.527617 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc0b40b5017859e1b0803ac831d27092d19986023b421294bf6e31b19f365e5d\": container with ID starting with fc0b40b5017859e1b0803ac831d27092d19986023b421294bf6e31b19f365e5d not found: ID does not exist" containerID="fc0b40b5017859e1b0803ac831d27092d19986023b421294bf6e31b19f365e5d" Oct 29 23:33:28.527703 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.527653 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc0b40b5017859e1b0803ac831d27092d19986023b421294bf6e31b19f365e5d"} err="failed to get container status \"fc0b40b5017859e1b0803ac831d27092d19986023b421294bf6e31b19f365e5d\": rpc error: code = NotFound desc = could not find container \"fc0b40b5017859e1b0803ac831d27092d19986023b421294bf6e31b19f365e5d\": container with ID starting with fc0b40b5017859e1b0803ac831d27092d19986023b421294bf6e31b19f365e5d not found: ID does not exist" Oct 29 23:33:28.527703 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.527674 5057 scope.go:122] "RemoveContainer" containerID="081524ad795f0d79617a18403c6e36507788c2a4e9441e0832a1e3ccfbb6fbd0" Oct 29 23:33:28.528025 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:33:28.527994 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"081524ad795f0d79617a18403c6e36507788c2a4e9441e0832a1e3ccfbb6fbd0\": container with ID starting with 081524ad795f0d79617a18403c6e36507788c2a4e9441e0832a1e3ccfbb6fbd0 not found: ID does not exist" containerID="081524ad795f0d79617a18403c6e36507788c2a4e9441e0832a1e3ccfbb6fbd0" Oct 29 23:33:28.528119 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.528027 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081524ad795f0d79617a18403c6e36507788c2a4e9441e0832a1e3ccfbb6fbd0"} err="failed to get container status \"081524ad795f0d79617a18403c6e36507788c2a4e9441e0832a1e3ccfbb6fbd0\": rpc error: code = NotFound desc = could not find container \"081524ad795f0d79617a18403c6e36507788c2a4e9441e0832a1e3ccfbb6fbd0\": container with ID starting with 081524ad795f0d79617a18403c6e36507788c2a4e9441e0832a1e3ccfbb6fbd0 not found: ID does not exist" Oct 29 23:33:28.561348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.561316 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh"] Oct 29 23:33:28.568442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:28.568412 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-b45gh"] Oct 29 23:33:29.734211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:29.721807 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c287175-5866-4c13-b806-5839582839cc" path="/var/lib/kubelet/pods/5c287175-5866-4c13-b806-5839582839cc/volumes" Oct 29 23:33:30.206806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.206771 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:33:30.207134 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.207106 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" podUID="c5db1599-6906-4dcd-8182-787026e2e745" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:33:30.727632 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.727593 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b"] Oct 29 23:33:30.729086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.729064 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5c287175-5866-4c13-b806-5839582839cc" containerName="istio-proxy" Oct 29 23:33:30.729247 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.729087 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c287175-5866-4c13-b806-5839582839cc" containerName="istio-proxy" Oct 29 23:33:30.729247 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.729157 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5c287175-5866-4c13-b806-5839582839cc" containerName="istio-proxy" Oct 29 23:33:30.729247 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.729165 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c287175-5866-4c13-b806-5839582839cc" containerName="istio-proxy" Oct 29 23:33:30.729419 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.729411 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c287175-5866-4c13-b806-5839582839cc" containerName="istio-proxy" Oct 29 23:33:30.735892 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.735866 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" Oct 29 23:33:30.763166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.763129 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b"] Oct 29 23:33:30.767900 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.767849 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e1769a\"/\"agc-cred-757501a65e29\"" Oct 29 23:33:30.849922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.849880 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq7gs\" (UniqueName: \"kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-kube-api-access-nq7gs\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b\" (UID: \"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" Oct 29 23:33:30.850343 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.850302 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b\" (UID: \"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" Oct 29 23:33:30.850499 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.850475 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b\" (UID: \"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" Oct 29 23:33:30.850723 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.850694 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b\" (UID: \"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" Oct 29 23:33:30.952203 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.952153 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b\" (UID: \"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" Oct 29 23:33:30.952417 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.952282 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b\" (UID: \"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" Oct 29 23:33:30.952417 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.952341 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b\" (UID: \"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" Oct 29 23:33:30.952417 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.952403 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq7gs\" (UniqueName: \"kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-kube-api-access-nq7gs\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b\" (UID: \"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" Oct 29 23:33:30.953113 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.953080 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b\" (UID: \"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" Oct 29 23:33:30.953367 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.953340 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b\" (UID: \"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" Oct 29 23:33:30.955312 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.955285 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b\" (UID: \"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" Oct 29 23:33:30.978086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:30.978009 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq7gs\" (UniqueName: \"kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-kube-api-access-nq7gs\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b\" (UID: \"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" Oct 29 23:33:31.056123 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.056084 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" Oct 29 23:33:31.377489 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:33:31.377432 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa921cc_cfbe_4194_9ca4_91c9f999cf1a.slice/crio-d94f25761fcaded38ff561e64ca793cee34203600d1c8d3625a517c6bcd2f140 WatchSource:0}: Error finding container d94f25761fcaded38ff561e64ca793cee34203600d1c8d3625a517c6bcd2f140: Status 404 returned error can't find the container with id d94f25761fcaded38ff561e64ca793cee34203600d1c8d3625a517c6bcd2f140 Oct 29 23:33:31.379611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.379582 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b"] Oct 29 23:33:31.469493 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.469466 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l"] Oct 29 23:33:31.470588 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.470569 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c287175-5866-4c13-b806-5839582839cc" containerName="istio-proxy" Oct 29 23:33:31.474754 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.474728 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" Oct 29 23:33:31.493429 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.493398 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-206095\"/\"agc-cred-873d12f62d54\"" Oct 29 23:33:31.499221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.499181 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l"] Oct 29 23:33:31.563140 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.563101 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l\" (UID: \"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" Oct 29 23:33:31.563358 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.563148 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8jq7\" (UniqueName: \"kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-kube-api-access-b8jq7\") pod \"payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l\" (UID: \"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" Oct 29 23:33:31.563414 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.563381 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l\" (UID: \"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" Oct 29 23:33:31.563528 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.563509 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l\" (UID: \"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" Oct 29 23:33:31.677689 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.677650 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l\" (UID: \"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" Oct 29 23:33:31.677914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.677888 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l\" (UID: \"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" Oct 29 23:33:31.678034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.677930 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8jq7\" (UniqueName: \"kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-kube-api-access-b8jq7\") pod \"payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l\" (UID: \"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" Oct 29 23:33:31.679160 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.678245 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l\" (UID: \"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" Oct 29 23:33:31.679160 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.678843 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l\" (UID: \"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" Oct 29 23:33:31.679160 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.679107 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l\" (UID: \"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" Oct 29 23:33:31.682301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.682267 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l\" (UID: \"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" Oct 29 23:33:31.738211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.735611 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8jq7\" (UniqueName: \"kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-kube-api-access-b8jq7\") pod \"payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l\" (UID: \"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" Oct 29 23:33:31.810210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.796461 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" Oct 29 23:33:31.905393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.905348 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" event={"ID":"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a","Type":"ContainerStarted","Data":"45253183432f835ee53b56c8f94b908947028e547fe421da82f4040e291b0efc"} Oct 29 23:33:31.905543 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.905412 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" event={"ID":"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a","Type":"ContainerStarted","Data":"d94f25761fcaded38ff561e64ca793cee34203600d1c8d3625a517c6bcd2f140"} Oct 29 23:33:31.906809 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.906778 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" Oct 29 23:33:31.907493 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.907472 5057 generic.go:356] "Generic (PLEG): container finished" podID="c5db1599-6906-4dcd-8182-787026e2e745" containerID="baa166cb6c5d9234be7485ef6531651d68a362859ac6d517c99c59e258748111" exitCode=0 Oct 29 23:33:31.907550 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.907515 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" event={"ID":"c5db1599-6906-4dcd-8182-787026e2e745","Type":"ContainerDied","Data":"baa166cb6c5d9234be7485ef6531651d68a362859ac6d517c99c59e258748111"} Oct 29 23:33:31.937037 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.937005 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 23:33:31.957662 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:31.957498 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" podStartSLOduration=1.957481952 podStartE2EDuration="1.957481952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:31.950701635 +0000 UTC m=+3058.602200166" watchObservedRunningTime="2026-10-29 23:33:31.957481952 +0000 UTC m=+3058.608980482" Oct 29 23:33:32.060906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.058365 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-istio-envoy\") pod \"c5db1599-6906-4dcd-8182-787026e2e745\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " Oct 29 23:33:32.060906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.058454 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-workload-socket\") pod \"c5db1599-6906-4dcd-8182-787026e2e745\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " Oct 29 23:33:32.060906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.058550 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-istio-data\") pod \"c5db1599-6906-4dcd-8182-787026e2e745\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " Oct 29 23:33:32.060906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.058674 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-workload-certs\") pod \"c5db1599-6906-4dcd-8182-787026e2e745\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " Oct 29 23:33:32.060906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.058770 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5db1599-6906-4dcd-8182-787026e2e745-istio-token\" (UniqueName: \"kubernetes.io/projected/c5db1599-6906-4dcd-8182-787026e2e745-istio-token\") pod \"c5db1599-6906-4dcd-8182-787026e2e745\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " Oct 29 23:33:32.060906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.058809 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-credential-socket\") pod \"c5db1599-6906-4dcd-8182-787026e2e745\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " Oct 29 23:33:32.060906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.058824 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-workload-socket" pod "c5db1599-6906-4dcd-8182-787026e2e745" (UID: "c5db1599-6906-4dcd-8182-787026e2e745"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:32.060906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.058903 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5db1599-6906-4dcd-8182-787026e2e745-kube-api-access-lvtbw\" (UniqueName: \"kubernetes.io/projected/c5db1599-6906-4dcd-8182-787026e2e745-kube-api-access-lvtbw\") pod \"c5db1599-6906-4dcd-8182-787026e2e745\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " Oct 29 23:33:32.060906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.058952 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5db1599-6906-4dcd-8182-787026e2e745-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5db1599-6906-4dcd-8182-787026e2e745-istiod-ca-cert\") pod \"c5db1599-6906-4dcd-8182-787026e2e745\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " Oct 29 23:33:32.060906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.059095 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-workload-certs" pod "c5db1599-6906-4dcd-8182-787026e2e745" (UID: "c5db1599-6906-4dcd-8182-787026e2e745"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:32.060906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.059198 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5db1599-6906-4dcd-8182-787026e2e745-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5db1599-6906-4dcd-8182-787026e2e745-istio-podinfo\") pod \"c5db1599-6906-4dcd-8182-787026e2e745\" (UID: \"c5db1599-6906-4dcd-8182-787026e2e745\") " Oct 29 23:33:32.060906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.059392 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-istio-data" pod "c5db1599-6906-4dcd-8182-787026e2e745" (UID: "c5db1599-6906-4dcd-8182-787026e2e745"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:32.060906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.059623 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-credential-socket" pod "c5db1599-6906-4dcd-8182-787026e2e745" (UID: "c5db1599-6906-4dcd-8182-787026e2e745"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:32.060906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.060199 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5db1599-6906-4dcd-8182-787026e2e745-istiod-ca-cert" pod "c5db1599-6906-4dcd-8182-787026e2e745" (UID: "c5db1599-6906-4dcd-8182-787026e2e745"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:32.060906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.060559 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:32.060906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.060580 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:32.060906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.060592 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:32.060906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.060606 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5db1599-6906-4dcd-8182-787026e2e745-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:32.063572 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.063539 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5db1599-6906-4dcd-8182-787026e2e745-istio-podinfo" pod "c5db1599-6906-4dcd-8182-787026e2e745" (UID: "c5db1599-6906-4dcd-8182-787026e2e745"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:33:32.063995 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.063970 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5db1599-6906-4dcd-8182-787026e2e745-kube-api-access-lvtbw" pod "c5db1599-6906-4dcd-8182-787026e2e745" (UID: "c5db1599-6906-4dcd-8182-787026e2e745"). InnerVolumeSpecName "kube-api-access-lvtbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:32.064291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.064255 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5db1599-6906-4dcd-8182-787026e2e745-istio-token" pod "c5db1599-6906-4dcd-8182-787026e2e745" (UID: "c5db1599-6906-4dcd-8182-787026e2e745"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:32.064965 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.064938 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-istio-envoy" pod "c5db1599-6906-4dcd-8182-787026e2e745" (UID: "c5db1599-6906-4dcd-8182-787026e2e745"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:32.161911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.161877 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:32.161911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.161907 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5db1599-6906-4dcd-8182-787026e2e745-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:32.161911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.161916 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5db1599-6906-4dcd-8182-787026e2e745-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:32.162156 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.161925 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvtbw\" (UniqueName: \"kubernetes.io/projected/c5db1599-6906-4dcd-8182-787026e2e745-kube-api-access-lvtbw\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:32.162156 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.161934 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5db1599-6906-4dcd-8182-787026e2e745-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:32.221609 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:33:32.221432 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1f5545_df3e_4bdf_b69d_dd4ef38d5072.slice/crio-f589eab0c5df0835e172f2b0eeb0c5eb46a66e70cea90f7aa0ddab5f589679b2 WatchSource:0}: Error finding container f589eab0c5df0835e172f2b0eeb0c5eb46a66e70cea90f7aa0ddab5f589679b2: Status 404 returned error can't find the container with id f589eab0c5df0835e172f2b0eeb0c5eb46a66e70cea90f7aa0ddab5f589679b2 Oct 29 23:33:32.222255 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.222204 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l"] Oct 29 23:33:32.918278 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.918241 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" event={"ID":"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072","Type":"ContainerStarted","Data":"dd08d5fcd850e163059930ee7b08b05e041a226d1a7043737d9ba87e5d94eba7"} Oct 29 23:33:32.918787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.918299 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" event={"ID":"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072","Type":"ContainerStarted","Data":"f589eab0c5df0835e172f2b0eeb0c5eb46a66e70cea90f7aa0ddab5f589679b2"} Oct 29 23:33:32.918787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.918353 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" Oct 29 23:33:32.919904 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.919873 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" event={"ID":"c5db1599-6906-4dcd-8182-787026e2e745","Type":"ContainerDied","Data":"4a30955dab50f8b988f4db608a74bf2950c0298f16515aecbf65f0762721802f"} Oct 29 23:33:32.920023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.919922 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb" Oct 29 23:33:32.920023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.919922 5057 scope.go:122] "RemoveContainer" containerID="baa166cb6c5d9234be7485ef6531651d68a362859ac6d517c99c59e258748111" Oct 29 23:33:32.971297 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:32.971232 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" podStartSLOduration=1.971213751 podStartE2EDuration="1.971213751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:32.962941295 +0000 UTC m=+3059.614439820" watchObservedRunningTime="2026-10-29 23:33:32.971213751 +0000 UTC m=+3059.622712278" Oct 29 23:33:33.006864 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:33.006831 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb"] Oct 29 23:33:33.024305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:33.024268 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-wpstb"] Oct 29 23:33:33.486701 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:33.486672 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5db1599-6906-4dcd-8182-787026e2e745" path="/var/lib/kubelet/pods/c5db1599-6906-4dcd-8182-787026e2e745/volumes" Oct 29 23:33:33.915960 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:33:33.915872 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:33:41.274003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:41.273962 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr"] Oct 29 23:33:44.034221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:44.033569 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" Oct 29 23:33:44.074224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:44.050419 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" Oct 29 23:33:44.581870 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:44.581834 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5"] Oct 29 23:33:44.610198 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:44.610147 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w"] Oct 29 23:33:45.484212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:45.481660 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd"] Oct 29 23:33:45.484212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:45.483115 5057 cpu_manager.go:414] "containerMap: removing container" podUID="c5db1599-6906-4dcd-8182-787026e2e745" containerName="istio-proxy" Oct 29 23:33:45.484212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:45.483141 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5db1599-6906-4dcd-8182-787026e2e745" containerName="istio-proxy" Oct 29 23:33:45.484212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:45.483530 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5db1599-6906-4dcd-8182-787026e2e745" containerName="istio-proxy" Oct 29 23:33:45.829092 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:45.829004 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" Oct 29 23:33:45.866211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:45.863036 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-45036b\"/\"payload-processing-external-model-e2e--86ce1f4b-dockercfg-jxqhv\"" Oct 29 23:33:45.884445 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:45.866507 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-45036b\"/\"agc-cred-ebd330ca5bd8\"" Oct 29 23:33:46.069868 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.069825 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv8hb\" (UniqueName: \"kubernetes.io/projected/df998551-ff23-4d28-b49d-c21dd71a9a7e-kube-api-access-pv8hb\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd\" (UID: \"df998551-ff23-4d28-b49d-c21dd71a9a7e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" Oct 29 23:33:46.102422 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.102326 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df998551-ff23-4d28-b49d-c21dd71a9a7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd\" (UID: \"df998551-ff23-4d28-b49d-c21dd71a9a7e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" Oct 29 23:33:46.137212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.102987 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df998551-ff23-4d28-b49d-c21dd71a9a7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd\" (UID: \"df998551-ff23-4d28-b49d-c21dd71a9a7e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" Oct 29 23:33:46.157373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.137685 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df998551-ff23-4d28-b49d-c21dd71a9a7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd\" (UID: \"df998551-ff23-4d28-b49d-c21dd71a9a7e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" Oct 29 23:33:46.168963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.168919 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd"] Oct 29 23:33:46.169210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.169182 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d"] Oct 29 23:33:46.174808 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.174779 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d"] Oct 29 23:33:46.174960 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.174866 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" Oct 29 23:33:46.191329 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.191297 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-45036b\"/\"agc-cred-15d4c16ca5d3\"" Oct 29 23:33:46.260092 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.260038 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6654e558-e0af-412c-94a3-5c958c854a4e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d\" (UID: \"6654e558-e0af-412c-94a3-5c958c854a4e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" Oct 29 23:33:46.260321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.260161 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df998551-ff23-4d28-b49d-c21dd71a9a7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd\" (UID: \"df998551-ff23-4d28-b49d-c21dd71a9a7e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" Oct 29 23:33:46.260399 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.260349 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df998551-ff23-4d28-b49d-c21dd71a9a7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd\" (UID: \"df998551-ff23-4d28-b49d-c21dd71a9a7e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" Oct 29 23:33:46.260465 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.260416 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv8hb\" (UniqueName: \"kubernetes.io/projected/df998551-ff23-4d28-b49d-c21dd71a9a7e-kube-api-access-pv8hb\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd\" (UID: \"df998551-ff23-4d28-b49d-c21dd71a9a7e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" Oct 29 23:33:46.260566 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.260513 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df998551-ff23-4d28-b49d-c21dd71a9a7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd\" (UID: \"df998551-ff23-4d28-b49d-c21dd71a9a7e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" Oct 29 23:33:46.260783 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.260758 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6654e558-e0af-412c-94a3-5c958c854a4e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d\" (UID: \"6654e558-e0af-412c-94a3-5c958c854a4e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" Oct 29 23:33:46.260888 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.260871 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6654e558-e0af-412c-94a3-5c958c854a4e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d\" (UID: \"6654e558-e0af-412c-94a3-5c958c854a4e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" Oct 29 23:33:46.261029 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.261009 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl68b\" (UniqueName: \"kubernetes.io/projected/6654e558-e0af-412c-94a3-5c958c854a4e-kube-api-access-tl68b\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d\" (UID: \"6654e558-e0af-412c-94a3-5c958c854a4e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" Oct 29 23:33:46.261239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.261214 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df998551-ff23-4d28-b49d-c21dd71a9a7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd\" (UID: \"df998551-ff23-4d28-b49d-c21dd71a9a7e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" Oct 29 23:33:46.261793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.261764 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df998551-ff23-4d28-b49d-c21dd71a9a7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd\" (UID: \"df998551-ff23-4d28-b49d-c21dd71a9a7e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" Oct 29 23:33:46.263799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.263771 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df998551-ff23-4d28-b49d-c21dd71a9a7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd\" (UID: \"df998551-ff23-4d28-b49d-c21dd71a9a7e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" Oct 29 23:33:46.274538 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.274501 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" podUID="ae6273c3-6af4-40fe-b208-8310228fc9a9" containerName="payload-processing" containerID="cri-o://5e79d78025b87ba0705ac9e20e246c730da274db82c7e912d3acdc2e9fefcd3d" gracePeriod=25 Oct 29 23:33:46.286259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.286233 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv8hb\" (UniqueName: \"kubernetes.io/projected/df998551-ff23-4d28-b49d-c21dd71a9a7e-kube-api-access-pv8hb\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd\" (UID: \"df998551-ff23-4d28-b49d-c21dd71a9a7e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" Oct 29 23:33:46.362949 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.362860 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6654e558-e0af-412c-94a3-5c958c854a4e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d\" (UID: \"6654e558-e0af-412c-94a3-5c958c854a4e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" Oct 29 23:33:46.363135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.362969 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6654e558-e0af-412c-94a3-5c958c854a4e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d\" (UID: \"6654e558-e0af-412c-94a3-5c958c854a4e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" Oct 29 23:33:46.363135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.363060 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl68b\" (UniqueName: \"kubernetes.io/projected/6654e558-e0af-412c-94a3-5c958c854a4e-kube-api-access-tl68b\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d\" (UID: \"6654e558-e0af-412c-94a3-5c958c854a4e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" Oct 29 23:33:46.363135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.363101 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6654e558-e0af-412c-94a3-5c958c854a4e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d\" (UID: \"6654e558-e0af-412c-94a3-5c958c854a4e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" Oct 29 23:33:46.363931 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.363875 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6654e558-e0af-412c-94a3-5c958c854a4e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d\" (UID: \"6654e558-e0af-412c-94a3-5c958c854a4e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" Oct 29 23:33:46.364086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.364031 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6654e558-e0af-412c-94a3-5c958c854a4e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d\" (UID: \"6654e558-e0af-412c-94a3-5c958c854a4e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" Oct 29 23:33:46.365835 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.365809 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6654e558-e0af-412c-94a3-5c958c854a4e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d\" (UID: \"6654e558-e0af-412c-94a3-5c958c854a4e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" Oct 29 23:33:46.388712 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.388674 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl68b\" (UniqueName: \"kubernetes.io/projected/6654e558-e0af-412c-94a3-5c958c854a4e-kube-api-access-tl68b\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d\" (UID: \"6654e558-e0af-412c-94a3-5c958c854a4e\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" Oct 29 23:33:46.423800 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.423763 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" Oct 29 23:33:46.494226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.492544 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" Oct 29 23:33:46.598291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.596264 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" Oct 29 23:33:46.678371 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.678325 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6273c3-6af4-40fe-b208-8310228fc9a9-kube-api-access-9wzfb\" (UniqueName: \"kubernetes.io/projected/ae6273c3-6af4-40fe-b208-8310228fc9a9-kube-api-access-9wzfb\") pod \"ae6273c3-6af4-40fe-b208-8310228fc9a9\" (UID: \"ae6273c3-6af4-40fe-b208-8310228fc9a9\") " Oct 29 23:33:46.678914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.678883 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6273c3-6af4-40fe-b208-8310228fc9a9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ae6273c3-6af4-40fe-b208-8310228fc9a9-provider-credentials\") pod \"ae6273c3-6af4-40fe-b208-8310228fc9a9\" (UID: \"ae6273c3-6af4-40fe-b208-8310228fc9a9\") " Oct 29 23:33:46.679351 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.679326 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae6273c3-6af4-40fe-b208-8310228fc9a9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ae6273c3-6af4-40fe-b208-8310228fc9a9-routing-overlay-volume\") pod \"ae6273c3-6af4-40fe-b208-8310228fc9a9\" (UID: \"ae6273c3-6af4-40fe-b208-8310228fc9a9\") " Oct 29 23:33:46.679743 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.679716 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6273c3-6af4-40fe-b208-8310228fc9a9-routing-overlay-volume" pod "ae6273c3-6af4-40fe-b208-8310228fc9a9" (UID: "ae6273c3-6af4-40fe-b208-8310228fc9a9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:46.680094 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.680072 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae6273c3-6af4-40fe-b208-8310228fc9a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae6273c3-6af4-40fe-b208-8310228fc9a9-plugins-config-volume\") pod \"ae6273c3-6af4-40fe-b208-8310228fc9a9\" (UID: \"ae6273c3-6af4-40fe-b208-8310228fc9a9\") " Oct 29 23:33:46.680561 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.680535 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6273c3-6af4-40fe-b208-8310228fc9a9-plugins-config-volume" pod "ae6273c3-6af4-40fe-b208-8310228fc9a9" (UID: "ae6273c3-6af4-40fe-b208-8310228fc9a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:46.681904 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.681615 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6273c3-6af4-40fe-b208-8310228fc9a9-provider-credentials" pod "ae6273c3-6af4-40fe-b208-8310228fc9a9" (UID: "ae6273c3-6af4-40fe-b208-8310228fc9a9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:46.681904 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.681721 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6273c3-6af4-40fe-b208-8310228fc9a9-kube-api-access-9wzfb" pod "ae6273c3-6af4-40fe-b208-8310228fc9a9" (UID: "ae6273c3-6af4-40fe-b208-8310228fc9a9"). InnerVolumeSpecName "kube-api-access-9wzfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:46.681904 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.681892 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wzfb\" (UniqueName: \"kubernetes.io/projected/ae6273c3-6af4-40fe-b208-8310228fc9a9-kube-api-access-9wzfb\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.682265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.681911 5057 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ae6273c3-6af4-40fe-b208-8310228fc9a9-provider-credentials\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.682265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.681926 5057 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ae6273c3-6af4-40fe-b208-8310228fc9a9-routing-overlay-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.682265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.681942 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae6273c3-6af4-40fe-b208-8310228fc9a9-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.858207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.843609 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd"] Oct 29 23:33:46.910437 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:33:46.910402 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6654e558_e0af_412c_94a3_5c958c854a4e.slice/crio-d194cb9949a626637282b8f836b744d342cd5a8e31b5cc1e260e7ca4a4c16b25 WatchSource:0}: Error finding container d194cb9949a626637282b8f836b744d342cd5a8e31b5cc1e260e7ca4a4c16b25: Status 404 returned error can't find the container with id d194cb9949a626637282b8f836b744d342cd5a8e31b5cc1e260e7ca4a4c16b25 Oct 29 23:33:46.910580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:46.910476 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d"] Oct 29 23:33:47.348372 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:47.348332 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" event={"ID":"df998551-ff23-4d28-b49d-c21dd71a9a7e","Type":"ContainerStarted","Data":"a5a4d917a91cadc119bb90e086c6b8f303bb3c5955984c63011a1b73f63ef600"} Oct 29 23:33:47.348576 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:47.348396 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" event={"ID":"df998551-ff23-4d28-b49d-c21dd71a9a7e","Type":"ContainerStarted","Data":"cc3630312c727af2d48e3f1d73b924350a43decd62dddd36f037d413d1345bd5"} Oct 29 23:33:47.348576 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:47.348426 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" Oct 29 23:33:47.350664 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:47.350631 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" event={"ID":"6654e558-e0af-412c-94a3-5c958c854a4e","Type":"ContainerStarted","Data":"8cc49af359974d3aa10aac026dac2e52c6182ecddbd6077b80b6c79ed29e2eee"} Oct 29 23:33:47.350830 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:47.350689 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" event={"ID":"6654e558-e0af-412c-94a3-5c958c854a4e","Type":"ContainerStarted","Data":"d194cb9949a626637282b8f836b744d342cd5a8e31b5cc1e260e7ca4a4c16b25"} Oct 29 23:33:47.350830 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:47.350719 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" Oct 29 23:33:47.352170 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:47.352142 5057 generic.go:356] "Generic (PLEG): container finished" podID="ae6273c3-6af4-40fe-b208-8310228fc9a9" containerID="5e79d78025b87ba0705ac9e20e246c730da274db82c7e912d3acdc2e9fefcd3d" exitCode=0 Oct 29 23:33:47.352297 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:47.352175 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" Oct 29 23:33:47.352297 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:47.352218 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" event={"ID":"ae6273c3-6af4-40fe-b208-8310228fc9a9","Type":"ContainerDied","Data":"5e79d78025b87ba0705ac9e20e246c730da274db82c7e912d3acdc2e9fefcd3d"} Oct 29 23:33:47.352297 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:47.352278 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr" event={"ID":"ae6273c3-6af4-40fe-b208-8310228fc9a9","Type":"ContainerDied","Data":"7ea1fed1639aaa29908dcc8326cef7c38f917e867ed6eea27704049bb669e86b"} Oct 29 23:33:47.352440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:47.352308 5057 scope.go:122] "RemoveContainer" containerID="5e79d78025b87ba0705ac9e20e246c730da274db82c7e912d3acdc2e9fefcd3d" Oct 29 23:33:47.364418 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:47.364399 5057 scope.go:122] "RemoveContainer" containerID="5e79d78025b87ba0705ac9e20e246c730da274db82c7e912d3acdc2e9fefcd3d" Oct 29 23:33:47.364806 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:33:47.364770 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e79d78025b87ba0705ac9e20e246c730da274db82c7e912d3acdc2e9fefcd3d\": container with ID starting with 5e79d78025b87ba0705ac9e20e246c730da274db82c7e912d3acdc2e9fefcd3d not found: ID does not exist" containerID="5e79d78025b87ba0705ac9e20e246c730da274db82c7e912d3acdc2e9fefcd3d" Oct 29 23:33:47.364894 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:47.364816 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e79d78025b87ba0705ac9e20e246c730da274db82c7e912d3acdc2e9fefcd3d"} err="failed to get container status \"5e79d78025b87ba0705ac9e20e246c730da274db82c7e912d3acdc2e9fefcd3d\": rpc error: code = NotFound desc = could not find container \"5e79d78025b87ba0705ac9e20e246c730da274db82c7e912d3acdc2e9fefcd3d\": container with ID starting with 5e79d78025b87ba0705ac9e20e246c730da274db82c7e912d3acdc2e9fefcd3d not found: ID does not exist" Oct 29 23:33:47.396339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:47.396284 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" podStartSLOduration=2.396265482 podStartE2EDuration="2.396265482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:47.392920273 +0000 UTC m=+3074.044418814" watchObservedRunningTime="2026-10-29 23:33:47.396265482 +0000 UTC m=+3074.047764025" Oct 29 23:33:47.442975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:47.442911 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" podStartSLOduration=2.442891771 podStartE2EDuration="2.442891771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:47.43795798 +0000 UTC m=+3074.089456516" watchObservedRunningTime="2026-10-29 23:33:47.442891771 +0000 UTC m=+3074.094390303" Oct 29 23:33:47.485422 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:47.485391 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr"] Oct 29 23:33:47.491688 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:47.491658 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-6986297vr"] Oct 29 23:33:49.485701 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:49.485666 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae6273c3-6af4-40fe-b208-8310228fc9a9" path="/var/lib/kubelet/pods/ae6273c3-6af4-40fe-b208-8310228fc9a9/volumes" Oct 29 23:33:49.583293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:49.583249 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" podUID="81b0acc4-fa4c-4596-8a30-262c5c1d26ac" containerName="payload-processing" containerID="cri-o://a56994617f21205a173f1f7d580f4669e41c00afedec8c1f497b53f1d2cfeee0" gracePeriod=25 Oct 29 23:33:49.611243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:49.611200 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" podUID="5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014" containerName="payload-processing" containerID="cri-o://831df0393a302fe6a31e87d440e349d39d3537901b5b595ddfb1f4b1b1c2c4a1" gracePeriod=25 Oct 29 23:33:49.960524 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:49.960338 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" Oct 29 23:33:49.999464 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:49.999438 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" Oct 29 23:33:50.023955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.023919 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-routing-overlay-volume\") pod \"81b0acc4-fa4c-4596-8a30-262c5c1d26ac\" (UID: \"81b0acc4-fa4c-4596-8a30-262c5c1d26ac\") " Oct 29 23:33:50.024224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.023985 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-routing-overlay-volume\") pod \"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014\" (UID: \"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014\") " Oct 29 23:33:50.024224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.024071 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-provider-credentials\" (UniqueName: \"kubernetes.io/projected/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-provider-credentials\") pod \"81b0acc4-fa4c-4596-8a30-262c5c1d26ac\" (UID: \"81b0acc4-fa4c-4596-8a30-262c5c1d26ac\") " Oct 29 23:33:50.024413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.024243 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-plugins-config-volume\") pod \"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014\" (UID: \"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014\") " Oct 29 23:33:50.024413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.024333 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-plugins-config-volume\") pod \"81b0acc4-fa4c-4596-8a30-262c5c1d26ac\" (UID: \"81b0acc4-fa4c-4596-8a30-262c5c1d26ac\") " Oct 29 23:33:50.024413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.024374 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-provider-credentials\") pod \"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014\" (UID: \"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014\") " Oct 29 23:33:50.024413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.024404 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-kube-api-access-9lln2\" (UniqueName: \"kubernetes.io/projected/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-kube-api-access-9lln2\") pod \"81b0acc4-fa4c-4596-8a30-262c5c1d26ac\" (UID: \"81b0acc4-fa4c-4596-8a30-262c5c1d26ac\") " Oct 29 23:33:50.024682 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.024570 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-routing-overlay-volume" pod "5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014" (UID: "5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:50.024749 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.024713 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-kube-api-access-98wvf\" (UniqueName: \"kubernetes.io/projected/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-kube-api-access-98wvf\") pod \"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014\" (UID: \"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014\") " Oct 29 23:33:50.025521 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.025230 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-plugins-config-volume" pod "5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014" (UID: "5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:50.025679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.025578 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-plugins-config-volume" pod "81b0acc4-fa4c-4596-8a30-262c5c1d26ac" (UID: "81b0acc4-fa4c-4596-8a30-262c5c1d26ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:50.026409 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.026329 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:50.026409 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.026357 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:50.026409 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.026374 5057 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-routing-overlay-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:50.026409 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.026327 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-routing-overlay-volume" pod "81b0acc4-fa4c-4596-8a30-262c5c1d26ac" (UID: "81b0acc4-fa4c-4596-8a30-262c5c1d26ac"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:50.041216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.037855 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-kube-api-access-9lln2" pod "81b0acc4-fa4c-4596-8a30-262c5c1d26ac" (UID: "81b0acc4-fa4c-4596-8a30-262c5c1d26ac"). InnerVolumeSpecName "kube-api-access-9lln2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:50.041216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.038280 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-provider-credentials" pod "81b0acc4-fa4c-4596-8a30-262c5c1d26ac" (UID: "81b0acc4-fa4c-4596-8a30-262c5c1d26ac"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:50.041216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.038522 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-provider-credentials" pod "5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014" (UID: "5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:50.041216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.038640 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-kube-api-access-98wvf" pod "5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014" (UID: "5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014"). InnerVolumeSpecName "kube-api-access-98wvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:50.128173 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.128083 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98wvf\" (UniqueName: \"kubernetes.io/projected/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-kube-api-access-98wvf\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:50.128173 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.128117 5057 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-routing-overlay-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:50.128173 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.128130 5057 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-provider-credentials\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:50.128173 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.128143 5057 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014-provider-credentials\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:50.128173 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.128155 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lln2\" (UniqueName: \"kubernetes.io/projected/81b0acc4-fa4c-4596-8a30-262c5c1d26ac-kube-api-access-9lln2\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:33:50.383698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.383618 5057 generic.go:356] "Generic (PLEG): container finished" podID="5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014" containerID="831df0393a302fe6a31e87d440e349d39d3537901b5b595ddfb1f4b1b1c2c4a1" exitCode=0 Oct 29 23:33:50.383698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.383672 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" Oct 29 23:33:50.383951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.383714 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" event={"ID":"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014","Type":"ContainerDied","Data":"831df0393a302fe6a31e87d440e349d39d3537901b5b595ddfb1f4b1b1c2c4a1"} Oct 29 23:33:50.383951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.383775 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w" event={"ID":"5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014","Type":"ContainerDied","Data":"e631ebec29b82cef6d962ac89aeef2792a0d43815ee8e0690a370bfd1bb06bdb"} Oct 29 23:33:50.383951 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.383807 5057 scope.go:122] "RemoveContainer" containerID="831df0393a302fe6a31e87d440e349d39d3537901b5b595ddfb1f4b1b1c2c4a1" Oct 29 23:33:50.385371 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.385349 5057 generic.go:356] "Generic (PLEG): container finished" podID="81b0acc4-fa4c-4596-8a30-262c5c1d26ac" containerID="a56994617f21205a173f1f7d580f4669e41c00afedec8c1f497b53f1d2cfeee0" exitCode=0 Oct 29 23:33:50.385697 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.385375 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" event={"ID":"81b0acc4-fa4c-4596-8a30-262c5c1d26ac","Type":"ContainerDied","Data":"a56994617f21205a173f1f7d580f4669e41c00afedec8c1f497b53f1d2cfeee0"} Oct 29 23:33:50.385697 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.385394 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" Oct 29 23:33:50.385697 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.385431 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5" event={"ID":"81b0acc4-fa4c-4596-8a30-262c5c1d26ac","Type":"ContainerDied","Data":"0e00e358c7a310ec4faee36bbb82b9b79d1dc0901873a2a026b8e757622a1da7"} Oct 29 23:33:50.396442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.396418 5057 scope.go:122] "RemoveContainer" containerID="831df0393a302fe6a31e87d440e349d39d3537901b5b595ddfb1f4b1b1c2c4a1" Oct 29 23:33:50.396735 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:33:50.396714 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"831df0393a302fe6a31e87d440e349d39d3537901b5b595ddfb1f4b1b1c2c4a1\": container with ID starting with 831df0393a302fe6a31e87d440e349d39d3537901b5b595ddfb1f4b1b1c2c4a1 not found: ID does not exist" containerID="831df0393a302fe6a31e87d440e349d39d3537901b5b595ddfb1f4b1b1c2c4a1" Oct 29 23:33:50.396828 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.396740 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"831df0393a302fe6a31e87d440e349d39d3537901b5b595ddfb1f4b1b1c2c4a1"} err="failed to get container status \"831df0393a302fe6a31e87d440e349d39d3537901b5b595ddfb1f4b1b1c2c4a1\": rpc error: code = NotFound desc = could not find container \"831df0393a302fe6a31e87d440e349d39d3537901b5b595ddfb1f4b1b1c2c4a1\": container with ID starting with 831df0393a302fe6a31e87d440e349d39d3537901b5b595ddfb1f4b1b1c2c4a1 not found: ID does not exist" Oct 29 23:33:50.396828 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.396759 5057 scope.go:122] "RemoveContainer" containerID="a56994617f21205a173f1f7d580f4669e41c00afedec8c1f497b53f1d2cfeee0" Oct 29 23:33:50.411768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.411741 5057 scope.go:122] "RemoveContainer" containerID="a56994617f21205a173f1f7d580f4669e41c00afedec8c1f497b53f1d2cfeee0" Oct 29 23:33:50.412095 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:33:50.412066 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56994617f21205a173f1f7d580f4669e41c00afedec8c1f497b53f1d2cfeee0\": container with ID starting with a56994617f21205a173f1f7d580f4669e41c00afedec8c1f497b53f1d2cfeee0 not found: ID does not exist" containerID="a56994617f21205a173f1f7d580f4669e41c00afedec8c1f497b53f1d2cfeee0" Oct 29 23:33:50.412211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.412105 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56994617f21205a173f1f7d580f4669e41c00afedec8c1f497b53f1d2cfeee0"} err="failed to get container status \"a56994617f21205a173f1f7d580f4669e41c00afedec8c1f497b53f1d2cfeee0\": rpc error: code = NotFound desc = could not find container \"a56994617f21205a173f1f7d580f4669e41c00afedec8c1f497b53f1d2cfeee0\": container with ID starting with a56994617f21205a173f1f7d580f4669e41c00afedec8c1f497b53f1d2cfeee0 not found: ID does not exist" Oct 29 23:33:50.482210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.472700 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w"] Oct 29 23:33:50.526430 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.526396 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-696666f4w"] Oct 29 23:33:50.563142 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.563109 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5"] Oct 29 23:33:50.569405 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:50.569369 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdts8c5"] Oct 29 23:33:51.485734 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:51.485701 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014" path="/var/lib/kubelet/pods/5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014/volumes" Oct 29 23:33:51.486253 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:51.486236 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81b0acc4-fa4c-4596-8a30-262c5c1d26ac" path="/var/lib/kubelet/pods/81b0acc4-fa4c-4596-8a30-262c5c1d26ac/volumes" Oct 29 23:33:56.105816 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.105776 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs"] Oct 29 23:33:56.107360 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.107332 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014" containerName="payload-processing" Oct 29 23:33:56.107360 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.107358 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014" containerName="payload-processing" Oct 29 23:33:56.107649 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.107386 5057 cpu_manager.go:414] "containerMap: removing container" podUID="81b0acc4-fa4c-4596-8a30-262c5c1d26ac" containerName="payload-processing" Oct 29 23:33:56.107649 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.107394 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b0acc4-fa4c-4596-8a30-262c5c1d26ac" containerName="payload-processing" Oct 29 23:33:56.107649 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.107428 5057 cpu_manager.go:414] "containerMap: removing container" podUID="ae6273c3-6af4-40fe-b208-8310228fc9a9" containerName="payload-processing" Oct 29 23:33:56.107649 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.107438 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae6273c3-6af4-40fe-b208-8310228fc9a9" containerName="payload-processing" Oct 29 23:33:56.107649 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.107637 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e0f1fc8-ff55-4ff1-b32d-0cd37e51d014" containerName="payload-processing" Oct 29 23:33:56.108032 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.107653 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b0acc4-fa4c-4596-8a30-262c5c1d26ac" containerName="payload-processing" Oct 29 23:33:56.108032 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.107667 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae6273c3-6af4-40fe-b208-8310228fc9a9" containerName="payload-processing" Oct 29 23:33:56.115524 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.114317 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.186814 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.183499 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6d9e36-openshift-default-dockercfg-9c6w9\"" Oct 29 23:33:56.291843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.212343 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs"] Oct 29 23:33:56.356265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.315166 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-workload-certs\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.488789 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.465462 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-envoy\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.488789 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.465538 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-data\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.535210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.490734 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istiod-ca-cert\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.535617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.535583 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-workload-socket\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.535900 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.535880 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2fbg\" (UniqueName: \"kubernetes.io/projected/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-kube-api-access-b2fbg\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.536152 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.536135 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-podinfo\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.536356 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.536337 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-credential-socket\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.536529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.536513 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-token\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.682001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.658623 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istiod-ca-cert\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.682001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.658689 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-workload-socket\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.682001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.658775 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2fbg\" (UniqueName: \"kubernetes.io/projected/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-kube-api-access-b2fbg\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.682001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.658871 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-podinfo\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.682001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.658920 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-credential-socket\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.682001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.658975 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-token\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.682001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.659006 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-workload-certs\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.682001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.659178 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-envoy\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.682001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.659235 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-data\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.682001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.659725 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-data\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.682001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.662885 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-workload-socket\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.682001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.663487 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-workload-certs\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.696845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.696719 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-credential-socket\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.698183 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.698145 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istiod-ca-cert\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.779418 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.774116 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-podinfo\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.811206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.806804 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-token\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.811206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.807644 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-envoy\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.811206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.808266 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2fbg\" (UniqueName: \"kubernetes.io/projected/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-kube-api-access-b2fbg\") pod \"e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:56.958056 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:56.957957 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:57.383647 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:33:57.383620 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03fd8f1b_0108_4471_a8bd_e1b98dbe8873.slice/crio-d564e69d6dd391e8ffbfe29a6c5ce4106375254d008d2e39c50195bed9423851 WatchSource:0}: Error finding container d564e69d6dd391e8ffbfe29a6c5ce4106375254d008d2e39c50195bed9423851: Status 404 returned error can't find the container with id d564e69d6dd391e8ffbfe29a6c5ce4106375254d008d2e39c50195bed9423851 Oct 29 23:33:57.384913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:57.384870 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs"] Oct 29 23:33:57.387860 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:57.387824 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:33:57.387956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:57.387911 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:33:57.388035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:57.387956 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:33:57.773945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:57.773905 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" event={"ID":"03fd8f1b-0108-4471-a8bd-e1b98dbe8873","Type":"ContainerStarted","Data":"0106c8f7f7cfe71fdfb41aa0e9bc475858acd8cde0c57ecc59c810a0ca037e2f"} Oct 29 23:33:57.774126 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:57.773968 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" event={"ID":"03fd8f1b-0108-4471-a8bd-e1b98dbe8873","Type":"ContainerStarted","Data":"d564e69d6dd391e8ffbfe29a6c5ce4106375254d008d2e39c50195bed9423851"} Oct 29 23:33:57.836213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:57.835301 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" podStartSLOduration=1.835280928 podStartE2EDuration="1.835280928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:57.819356969 +0000 UTC m=+3084.470855498" watchObservedRunningTime="2026-10-29 23:33:57.835280928 +0000 UTC m=+3084.486779460" Oct 29 23:33:57.968213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:57.958952 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:58.366520 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:58.366479 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" Oct 29 23:33:58.366720 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:58.366542 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" podUID="df998551-ff23-4d28-b49d-c21dd71a9a7e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:33:58.458946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:58.458910 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd"] Oct 29 23:33:58.461322 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:58.461289 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" podUID="df998551-ff23-4d28-b49d-c21dd71a9a7e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:33:58.966324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:58.966289 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:59.794582 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:59.794547 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:33:59.795380 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:33:59.795361 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:34:03.459949 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.459853 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" podUID="df998551-ff23-4d28-b49d-c21dd71a9a7e" containerName="payload-processing" containerID="cri-o://a5a4d917a91cadc119bb90e086c6b8f303bb3c5955984c63011a1b73f63ef600" gracePeriod=25 Oct 29 23:34:03.805351 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.805319 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" Oct 29 23:34:03.834960 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.834920 5057 generic.go:356] "Generic (PLEG): container finished" podID="8645590d-04a6-4cff-a65e-281d5cb7f059" containerID="4c3c6e40a4cae75758e4b48e4f060d61288fe485a7a8bb2d757735edf820d597" exitCode=0 Oct 29 23:34:03.835167 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.834983 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" event={"ID":"8645590d-04a6-4cff-a65e-281d5cb7f059","Type":"ContainerDied","Data":"4c3c6e40a4cae75758e4b48e4f060d61288fe485a7a8bb2d757735edf820d597"} Oct 29 23:34:03.835919 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.835876 5057 scope.go:122] "RemoveContainer" containerID="4c3c6e40a4cae75758e4b48e4f060d61288fe485a7a8bb2d757735edf820d597" Oct 29 23:34:03.837102 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.837055 5057 generic.go:356] "Generic (PLEG): container finished" podID="df998551-ff23-4d28-b49d-c21dd71a9a7e" containerID="a5a4d917a91cadc119bb90e086c6b8f303bb3c5955984c63011a1b73f63ef600" exitCode=0 Oct 29 23:34:03.837257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.837126 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" Oct 29 23:34:03.837257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.837148 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" event={"ID":"df998551-ff23-4d28-b49d-c21dd71a9a7e","Type":"ContainerDied","Data":"a5a4d917a91cadc119bb90e086c6b8f303bb3c5955984c63011a1b73f63ef600"} Oct 29 23:34:03.837398 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.837269 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd" event={"ID":"df998551-ff23-4d28-b49d-c21dd71a9a7e","Type":"ContainerDied","Data":"cc3630312c727af2d48e3f1d73b924350a43decd62dddd36f037d413d1345bd5"} Oct 29 23:34:03.837398 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.837304 5057 scope.go:122] "RemoveContainer" containerID="a5a4d917a91cadc119bb90e086c6b8f303bb3c5955984c63011a1b73f63ef600" Oct 29 23:34:03.838901 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.838712 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:34:03.838901 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.838787 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:34:03.838901 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.838822 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:34:03.863872 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.863841 5057 scope.go:122] "RemoveContainer" containerID="a5a4d917a91cadc119bb90e086c6b8f303bb3c5955984c63011a1b73f63ef600" Oct 29 23:34:03.865319 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:34:03.865281 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a4d917a91cadc119bb90e086c6b8f303bb3c5955984c63011a1b73f63ef600\": container with ID starting with a5a4d917a91cadc119bb90e086c6b8f303bb3c5955984c63011a1b73f63ef600 not found: ID does not exist" containerID="a5a4d917a91cadc119bb90e086c6b8f303bb3c5955984c63011a1b73f63ef600" Oct 29 23:34:03.865459 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.865326 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a4d917a91cadc119bb90e086c6b8f303bb3c5955984c63011a1b73f63ef600"} err="failed to get container status \"a5a4d917a91cadc119bb90e086c6b8f303bb3c5955984c63011a1b73f63ef600\": rpc error: code = NotFound desc = could not find container \"a5a4d917a91cadc119bb90e086c6b8f303bb3c5955984c63011a1b73f63ef600\": container with ID starting with a5a4d917a91cadc119bb90e086c6b8f303bb3c5955984c63011a1b73f63ef600 not found: ID does not exist" Oct 29 23:34:03.928703 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.925263 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df998551-ff23-4d28-b49d-c21dd71a9a7e-kube-api-access-pv8hb\" (UniqueName: \"kubernetes.io/projected/df998551-ff23-4d28-b49d-c21dd71a9a7e-kube-api-access-pv8hb\") pod \"df998551-ff23-4d28-b49d-c21dd71a9a7e\" (UID: \"df998551-ff23-4d28-b49d-c21dd71a9a7e\") " Oct 29 23:34:03.928703 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.925376 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df998551-ff23-4d28-b49d-c21dd71a9a7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df998551-ff23-4d28-b49d-c21dd71a9a7e-plugins-config-volume\") pod \"df998551-ff23-4d28-b49d-c21dd71a9a7e\" (UID: \"df998551-ff23-4d28-b49d-c21dd71a9a7e\") " Oct 29 23:34:03.928703 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.925451 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df998551-ff23-4d28-b49d-c21dd71a9a7e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df998551-ff23-4d28-b49d-c21dd71a9a7e-routing-overlay-volume\") pod \"df998551-ff23-4d28-b49d-c21dd71a9a7e\" (UID: \"df998551-ff23-4d28-b49d-c21dd71a9a7e\") " Oct 29 23:34:03.928703 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.925639 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df998551-ff23-4d28-b49d-c21dd71a9a7e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/df998551-ff23-4d28-b49d-c21dd71a9a7e-provider-credentials\") pod \"df998551-ff23-4d28-b49d-c21dd71a9a7e\" (UID: \"df998551-ff23-4d28-b49d-c21dd71a9a7e\") " Oct 29 23:34:03.933559 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.932460 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df998551-ff23-4d28-b49d-c21dd71a9a7e-plugins-config-volume" pod "df998551-ff23-4d28-b49d-c21dd71a9a7e" (UID: "df998551-ff23-4d28-b49d-c21dd71a9a7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:34:03.933559 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:34:03.932978 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:34:03.941960 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.935389 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df998551-ff23-4d28-b49d-c21dd71a9a7e-routing-overlay-volume" pod "df998551-ff23-4d28-b49d-c21dd71a9a7e" (UID: "df998551-ff23-4d28-b49d-c21dd71a9a7e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:34:03.950922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.950885 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df998551-ff23-4d28-b49d-c21dd71a9a7e-kube-api-access-pv8hb" pod "df998551-ff23-4d28-b49d-c21dd71a9a7e" (UID: "df998551-ff23-4d28-b49d-c21dd71a9a7e"). InnerVolumeSpecName "kube-api-access-pv8hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:34:03.954280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:03.954249 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df998551-ff23-4d28-b49d-c21dd71a9a7e-provider-credentials" pod "df998551-ff23-4d28-b49d-c21dd71a9a7e" (UID: "df998551-ff23-4d28-b49d-c21dd71a9a7e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:34:04.047753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:04.031953 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df998551-ff23-4d28-b49d-c21dd71a9a7e-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:34:04.047753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:04.031998 5057 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df998551-ff23-4d28-b49d-c21dd71a9a7e-routing-overlay-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:34:04.047753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:04.032014 5057 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df998551-ff23-4d28-b49d-c21dd71a9a7e-provider-credentials\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:34:04.047753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:04.032028 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv8hb\" (UniqueName: \"kubernetes.io/projected/df998551-ff23-4d28-b49d-c21dd71a9a7e-kube-api-access-pv8hb\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:34:04.191003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:04.190963 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd"] Oct 29 23:34:04.195940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:04.195908 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-84968ztsd"] Oct 29 23:34:04.871333 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:04.870661 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" event={"ID":"8645590d-04a6-4cff-a65e-281d5cb7f059","Type":"ContainerStarted","Data":"3fa14b2805b49b0e8faf99ea4ab05b38ab3b8ec105d0be50397a543aa65a8aa4"} Oct 29 23:34:05.487113 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:05.487080 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df998551-ff23-4d28-b49d-c21dd71a9a7e" path="/var/lib/kubelet/pods/df998551-ff23-4d28-b49d-c21dd71a9a7e/volumes" Oct 29 23:34:05.493696 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:05.493673 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:34:05.494808 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:05.494787 5057 patch_prober.go:29] interesting pod/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 29 23:34:05.494864 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:05.494831 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" podUID="8645590d-04a6-4cff-a65e-281d5cb7f059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 29 23:34:05.575911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:05.575871 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:34:06.493799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:06.493765 5057 patch_prober.go:29] interesting pod/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 29 23:34:06.494214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:06.493817 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" podUID="8645590d-04a6-4cff-a65e-281d5cb7f059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 29 23:34:07.494312 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:07.494276 5057 patch_prober.go:29] interesting pod/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 29 23:34:07.494729 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:07.494342 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" podUID="8645590d-04a6-4cff-a65e-281d5cb7f059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 29 23:34:08.494973 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:08.493725 5057 patch_prober.go:29] interesting pod/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 29 23:34:08.494973 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:08.493784 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" podUID="8645590d-04a6-4cff-a65e-281d5cb7f059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 29 23:34:09.494412 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:09.494379 5057 patch_prober.go:29] interesting pod/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 29 23:34:09.494674 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:09.494645 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" podUID="8645590d-04a6-4cff-a65e-281d5cb7f059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 29 23:34:10.499911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:10.499879 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:34:10.501165 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:10.501143 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:34:12.347618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:12.347579 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm"] Oct 29 23:34:12.349091 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:12.349064 5057 cpu_manager.go:414] "containerMap: removing container" podUID="df998551-ff23-4d28-b49d-c21dd71a9a7e" containerName="payload-processing" Oct 29 23:34:12.349091 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:12.349091 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df998551-ff23-4d28-b49d-c21dd71a9a7e" containerName="payload-processing" Oct 29 23:34:12.349349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:12.349336 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="df998551-ff23-4d28-b49d-c21dd71a9a7e" containerName="payload-processing" Oct 29 23:34:12.353805 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:12.353782 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm" Oct 29 23:34:12.377506 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:12.377480 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6d9e36\"" Oct 29 23:34:12.378062 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:12.378039 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6d9e36-dockercfg-w56cw\"" Oct 29 23:34:12.389151 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:12.389124 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm"] Oct 29 23:34:12.659266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:12.650669 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w"] Oct 29 23:34:12.767245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:12.756727 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/236cd687-b020-40e9-a69d-7acfcfe2b658-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm\" (UID: \"236cd687-b020-40e9-a69d-7acfcfe2b658\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm" Oct 29 23:34:12.767245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:12.756781 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5xjq\" (UniqueName: \"kubernetes.io/projected/236cd687-b020-40e9-a69d-7acfcfe2b658-kube-api-access-l5xjq\") pod \"payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm\" (UID: \"236cd687-b020-40e9-a69d-7acfcfe2b658\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm" Oct 29 23:34:13.000438 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:13.000396 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/236cd687-b020-40e9-a69d-7acfcfe2b658-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm\" (UID: \"236cd687-b020-40e9-a69d-7acfcfe2b658\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm" Oct 29 23:34:13.000659 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:13.000454 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5xjq\" (UniqueName: \"kubernetes.io/projected/236cd687-b020-40e9-a69d-7acfcfe2b658-kube-api-access-l5xjq\") pod \"payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm\" (UID: \"236cd687-b020-40e9-a69d-7acfcfe2b658\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm" Oct 29 23:34:13.003408 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:13.003375 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/236cd687-b020-40e9-a69d-7acfcfe2b658-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm\" (UID: \"236cd687-b020-40e9-a69d-7acfcfe2b658\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm" Oct 29 23:34:13.015653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:13.015620 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w"] Oct 29 23:34:13.016056 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:13.016036 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w" Oct 29 23:34:13.034178 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:13.034147 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5xjq\" (UniqueName: \"kubernetes.io/projected/236cd687-b020-40e9-a69d-7acfcfe2b658-kube-api-access-l5xjq\") pod \"payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm\" (UID: \"236cd687-b020-40e9-a69d-7acfcfe2b658\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm" Oct 29 23:34:13.102311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:13.102277 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w\" (UID: \"1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w" Oct 29 23:34:13.102482 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:13.102408 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnppc\" (UniqueName: \"kubernetes.io/projected/1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3-kube-api-access-xnppc\") pod \"payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w\" (UID: \"1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w" Oct 29 23:34:13.204288 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:13.204252 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w\" (UID: \"1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w" Oct 29 23:34:13.204456 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:13.204387 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnppc\" (UniqueName: \"kubernetes.io/projected/1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3-kube-api-access-xnppc\") pod \"payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w\" (UID: \"1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w" Oct 29 23:34:13.204960 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:13.204933 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w\" (UID: \"1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w" Oct 29 23:34:13.229747 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:13.229716 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnppc\" (UniqueName: \"kubernetes.io/projected/1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3-kube-api-access-xnppc\") pod \"payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w\" (UID: \"1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w" Oct 29 23:34:13.269131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:13.269049 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm" Oct 29 23:34:13.366367 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:13.366132 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w" Oct 29 23:34:13.593752 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:13.593714 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm"] Oct 29 23:34:13.593907 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:34:13.593889 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236cd687_b020_40e9_a69d_7acfcfe2b658.slice/crio-132cf1e863e7013d3ca0e6b7c2b7c5ffd16d901ea530c607db50883ee28f7328 WatchSource:0}: Error finding container 132cf1e863e7013d3ca0e6b7c2b7c5ffd16d901ea530c607db50883ee28f7328: Status 404 returned error can't find the container with id 132cf1e863e7013d3ca0e6b7c2b7c5ffd16d901ea530c607db50883ee28f7328 Oct 29 23:34:13.690723 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:34:13.690693 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6d1edb_6a64_45e0_96f4_b77ea25ebbf3.slice/crio-2b278b2bd00f2ffd0f83ff0e1ac12a748eb3394fd602bfc1c4c109ac7be28647 WatchSource:0}: Error finding container 2b278b2bd00f2ffd0f83ff0e1ac12a748eb3394fd602bfc1c4c109ac7be28647: Status 404 returned error can't find the container with id 2b278b2bd00f2ffd0f83ff0e1ac12a748eb3394fd602bfc1c4c109ac7be28647 Oct 29 23:34:13.694088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:13.694058 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w"] Oct 29 23:34:14.015374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:14.015340 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w" event={"ID":"1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3","Type":"ContainerStarted","Data":"2569d769ae96ae35a4402bfb282a353d28c4598e0a4c362ef92c08af5ae54408"} Oct 29 23:34:14.015576 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:14.015399 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w" event={"ID":"1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3","Type":"ContainerStarted","Data":"2b278b2bd00f2ffd0f83ff0e1ac12a748eb3394fd602bfc1c4c109ac7be28647"} Oct 29 23:34:14.015576 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:14.015447 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w" Oct 29 23:34:14.017200 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:14.017158 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm" event={"ID":"236cd687-b020-40e9-a69d-7acfcfe2b658","Type":"ContainerStarted","Data":"f2b738c55d8a6c70ad81fffd95885844e5541fc072be53b0a9de799af5693fa3"} Oct 29 23:34:14.017342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:14.017219 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm" event={"ID":"236cd687-b020-40e9-a69d-7acfcfe2b658","Type":"ContainerStarted","Data":"132cf1e863e7013d3ca0e6b7c2b7c5ffd16d901ea530c607db50883ee28f7328"} Oct 29 23:34:14.017342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:14.017253 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm" Oct 29 23:34:14.137462 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:14.133745 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w" podStartSLOduration=2.133723716 podStartE2EDuration="2.133723716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:34:14.063510562 +0000 UTC m=+3100.715009094" watchObservedRunningTime="2026-10-29 23:34:14.133723716 +0000 UTC m=+3100.785222249" Oct 29 23:34:14.137462 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:14.136156 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm" podStartSLOduration=2.136140409 podStartE2EDuration="2.136140409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:34:14.123422776 +0000 UTC m=+3100.774921320" watchObservedRunningTime="2026-10-29 23:34:14.136140409 +0000 UTC m=+3100.787638940" Oct 29 23:34:25.030519 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:25.030412 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w" Oct 29 23:34:25.031152 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:25.030889 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm" Oct 29 23:34:33.916073 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:34:33.916034 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:34:51.902797 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:51.901072 5057 patch_prober.go:29] interesting pod/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": read tcp 10.132.0.2:33026->10.132.1.29:15021: read: connection reset by peer" start-of-body= Oct 29 23:34:51.902797 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:51.901168 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" podUID="1729a889-e3d1-43df-89d1-f3356bc34a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": read tcp 10.132.0.2:33026->10.132.1.29:15021: read: connection reset by peer" Oct 29 23:34:52.888025 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:52.887993 5057 generic.go:356] "Generic (PLEG): container finished" podID="1729a889-e3d1-43df-89d1-f3356bc34a46" containerID="233a3bc3f3ec7b97f979dc0789e6e4b2cf6c87a2b201fcec7d6967f4e0bc98c5" exitCode=0 Oct 29 23:34:52.888253 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:52.888025 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" event={"ID":"1729a889-e3d1-43df-89d1-f3356bc34a46","Type":"ContainerDied","Data":"233a3bc3f3ec7b97f979dc0789e6e4b2cf6c87a2b201fcec7d6967f4e0bc98c5"} Oct 29 23:34:52.888544 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:52.888527 5057 scope.go:122] "RemoveContainer" containerID="233a3bc3f3ec7b97f979dc0789e6e4b2cf6c87a2b201fcec7d6967f4e0bc98c5" Oct 29 23:34:52.890869 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:52.890835 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:34:52.890997 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:52.890920 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:34:52.890997 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:52.890968 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:34:53.898854 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:53.898820 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" event={"ID":"1729a889-e3d1-43df-89d1-f3356bc34a46","Type":"ContainerStarted","Data":"2df8e5154fbbae8711f860b55eb2af6e5b9efed71a987260cdaf8acbccc3f755"} Oct 29 23:34:53.937998 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:53.937967 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:34:54.929210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:54.927664 5057 patch_prober.go:29] interesting pod/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Oct 29 23:34:54.929210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:54.927723 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" podUID="1729a889-e3d1-43df-89d1-f3356bc34a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Oct 29 23:34:55.948738 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:55.948292 5057 patch_prober.go:29] interesting pod/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Oct 29 23:34:55.948738 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:55.948372 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" podUID="1729a889-e3d1-43df-89d1-f3356bc34a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Oct 29 23:34:56.932212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:56.931301 5057 patch_prober.go:29] interesting pod/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Oct 29 23:34:56.932212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:56.931364 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" podUID="1729a889-e3d1-43df-89d1-f3356bc34a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Oct 29 23:34:57.939778 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:57.939321 5057 patch_prober.go:29] interesting pod/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Oct 29 23:34:57.939778 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:57.939385 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" podUID="1729a889-e3d1-43df-89d1-f3356bc34a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Oct 29 23:34:58.933219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:58.932294 5057 patch_prober.go:29] interesting pod/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Oct 29 23:34:58.933219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:58.932368 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" podUID="1729a889-e3d1-43df-89d1-f3356bc34a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Oct 29 23:34:59.931653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:59.931620 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:34:59.932244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:59.932221 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:34:59.933087 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:34:59.933066 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:35:01.668029 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:01.667985 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt"] Oct 29 23:35:01.674150 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:01.674122 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt" Oct 29 23:35:01.690090 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:01.690058 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b7a567\"" Oct 29 23:35:01.693287 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:01.693259 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt"] Oct 29 23:35:01.734417 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:01.734374 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c68413-7dbd-4108-ad58-37599bbe9f8e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt\" (UID: \"f2c68413-7dbd-4108-ad58-37599bbe9f8e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt" Oct 29 23:35:01.734630 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:01.734496 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm6s9\" (UniqueName: \"kubernetes.io/projected/f2c68413-7dbd-4108-ad58-37599bbe9f8e-kube-api-access-xm6s9\") pod \"payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt\" (UID: \"f2c68413-7dbd-4108-ad58-37599bbe9f8e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt" Oct 29 23:35:01.836733 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:01.836694 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c68413-7dbd-4108-ad58-37599bbe9f8e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt\" (UID: \"f2c68413-7dbd-4108-ad58-37599bbe9f8e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt" Oct 29 23:35:01.836933 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:01.836831 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm6s9\" (UniqueName: \"kubernetes.io/projected/f2c68413-7dbd-4108-ad58-37599bbe9f8e-kube-api-access-xm6s9\") pod \"payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt\" (UID: \"f2c68413-7dbd-4108-ad58-37599bbe9f8e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt" Oct 29 23:35:01.837604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:01.837575 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c68413-7dbd-4108-ad58-37599bbe9f8e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt\" (UID: \"f2c68413-7dbd-4108-ad58-37599bbe9f8e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt" Oct 29 23:35:01.861275 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:01.861240 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm6s9\" (UniqueName: \"kubernetes.io/projected/f2c68413-7dbd-4108-ad58-37599bbe9f8e-kube-api-access-xm6s9\") pod \"payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt\" (UID: \"f2c68413-7dbd-4108-ad58-37599bbe9f8e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt" Oct 29 23:35:01.991472 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:01.991160 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt" Oct 29 23:35:02.661798 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:02.661766 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt"] Oct 29 23:35:02.662458 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:35:02.662433 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c68413_7dbd_4108_ad58_37599bbe9f8e.slice/crio-19aa93517f4c3db87039261b5a793ef934bad615c85f01a427e58715162c3ca4 WatchSource:0}: Error finding container 19aa93517f4c3db87039261b5a793ef934bad615c85f01a427e58715162c3ca4: Status 404 returned error can't find the container with id 19aa93517f4c3db87039261b5a793ef934bad615c85f01a427e58715162c3ca4 Oct 29 23:35:03.211629 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:03.211574 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt" event={"ID":"f2c68413-7dbd-4108-ad58-37599bbe9f8e","Type":"ContainerStarted","Data":"b29f7735ef6899bbd393f193edebcdf14a0f33ae9ce763bfe95b727a334da0be"} Oct 29 23:35:03.212159 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:03.211643 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt" event={"ID":"f2c68413-7dbd-4108-ad58-37599bbe9f8e","Type":"ContainerStarted","Data":"19aa93517f4c3db87039261b5a793ef934bad615c85f01a427e58715162c3ca4"} Oct 29 23:35:03.212159 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:03.211733 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt" Oct 29 23:35:03.255348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:03.255291 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt" podStartSLOduration=2.255276462 podStartE2EDuration="2.255276462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:35:03.251853037 +0000 UTC m=+3149.903351591" watchObservedRunningTime="2026-10-29 23:35:03.255276462 +0000 UTC m=+3149.906774991" Oct 29 23:35:03.916156 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:35:03.916120 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:35:14.225087 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:14.225055 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt" Oct 29 23:35:26.920952 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:26.914635 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:35:30.454173 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:30.454132 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps"] Oct 29 23:35:30.454707 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:30.454451 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" podUID="9a4615b1-ff3d-4339-853a-f5d5247a5573" containerName="istio-proxy" containerID="cri-o://0957fa92c5d38be4195b35dc646131ab9e551749a298ff80df8f67070b262e71" gracePeriod=30 Oct 29 23:35:32.124846 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:32.124809 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:35:32.125503 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:32.124866 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" podUID="9a4615b1-ff3d-4339-853a-f5d5247a5573" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:35:33.915613 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:35:33.915572 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:35:35.793234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:35.793145 5057 generic.go:356] "Generic (PLEG): container finished" podID="9a4615b1-ff3d-4339-853a-f5d5247a5573" containerID="0957fa92c5d38be4195b35dc646131ab9e551749a298ff80df8f67070b262e71" exitCode=0 Oct 29 23:35:35.793747 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:35.793378 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" event={"ID":"9a4615b1-ff3d-4339-853a-f5d5247a5573","Type":"ContainerDied","Data":"0957fa92c5d38be4195b35dc646131ab9e551749a298ff80df8f67070b262e71"} Oct 29 23:35:35.793858 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:35.793829 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" event={"ID":"9a4615b1-ff3d-4339-853a-f5d5247a5573","Type":"ContainerDied","Data":"a12fb4555f10562286caac8eb77e90e5213b66abb1fe7466eb5002a166a6eea4"} Oct 29 23:35:35.793937 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:35.793856 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a12fb4555f10562286caac8eb77e90e5213b66abb1fe7466eb5002a166a6eea4" Oct 29 23:35:35.798468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:35.798447 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 23:35:36.021266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:35.995233 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-data\") pod \"9a4615b1-ff3d-4339-853a-f5d5247a5573\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " Oct 29 23:35:36.021266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:35.995295 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a4615b1-ff3d-4339-853a-f5d5247a5573-kube-api-access-g97wl\" (UniqueName: \"kubernetes.io/projected/9a4615b1-ff3d-4339-853a-f5d5247a5573-kube-api-access-g97wl\") pod \"9a4615b1-ff3d-4339-853a-f5d5247a5573\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " Oct 29 23:35:36.021266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:35.995333 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-envoy\") pod \"9a4615b1-ff3d-4339-853a-f5d5247a5573\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " Oct 29 23:35:36.021266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:35.995388 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-token\" (UniqueName: \"kubernetes.io/projected/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-token\") pod \"9a4615b1-ff3d-4339-853a-f5d5247a5573\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " Oct 29 23:35:36.021266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:35.995416 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-workload-certs\") pod \"9a4615b1-ff3d-4339-853a-f5d5247a5573\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " Oct 29 23:35:36.021266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:35.995479 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-podinfo\") pod \"9a4615b1-ff3d-4339-853a-f5d5247a5573\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " Oct 29 23:35:36.021266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:35.995561 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-credential-socket\") pod \"9a4615b1-ff3d-4339-853a-f5d5247a5573\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " Oct 29 23:35:36.021266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:35.995647 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a4615b1-ff3d-4339-853a-f5d5247a5573-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a4615b1-ff3d-4339-853a-f5d5247a5573-istiod-ca-cert\") pod \"9a4615b1-ff3d-4339-853a-f5d5247a5573\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " Oct 29 23:35:36.021266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:35.995934 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-workload-socket\") pod \"9a4615b1-ff3d-4339-853a-f5d5247a5573\" (UID: \"9a4615b1-ff3d-4339-853a-f5d5247a5573\") " Oct 29 23:35:36.056235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.054388 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-token" pod "9a4615b1-ff3d-4339-853a-f5d5247a5573" (UID: "9a4615b1-ff3d-4339-853a-f5d5247a5573"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:35:36.056593 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.056561 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-workload-certs" pod "9a4615b1-ff3d-4339-853a-f5d5247a5573" (UID: "9a4615b1-ff3d-4339-853a-f5d5247a5573"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:35:36.057638 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.057607 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-envoy" pod "9a4615b1-ff3d-4339-853a-f5d5247a5573" (UID: "9a4615b1-ff3d-4339-853a-f5d5247a5573"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:35:36.057779 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.057653 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-workload-socket" pod "9a4615b1-ff3d-4339-853a-f5d5247a5573" (UID: "9a4615b1-ff3d-4339-853a-f5d5247a5573"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:35:36.057986 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.057961 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a4615b1-ff3d-4339-853a-f5d5247a5573-kube-api-access-g97wl" pod "9a4615b1-ff3d-4339-853a-f5d5247a5573" (UID: "9a4615b1-ff3d-4339-853a-f5d5247a5573"). InnerVolumeSpecName "kube-api-access-g97wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:35:36.058076 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.057980 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-credential-socket" pod "9a4615b1-ff3d-4339-853a-f5d5247a5573" (UID: "9a4615b1-ff3d-4339-853a-f5d5247a5573"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:35:36.058076 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.057965 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-data" pod "9a4615b1-ff3d-4339-853a-f5d5247a5573" (UID: "9a4615b1-ff3d-4339-853a-f5d5247a5573"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:35:36.058868 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.058324 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a4615b1-ff3d-4339-853a-f5d5247a5573-istiod-ca-cert" pod "9a4615b1-ff3d-4339-853a-f5d5247a5573" (UID: "9a4615b1-ff3d-4339-853a-f5d5247a5573"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:35:36.060637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.060118 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-podinfo" pod "9a4615b1-ff3d-4339-853a-f5d5247a5573" (UID: "9a4615b1-ff3d-4339-853a-f5d5247a5573"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:35:36.149139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.149107 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:35:36.149139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.149138 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g97wl\" (UniqueName: \"kubernetes.io/projected/9a4615b1-ff3d-4339-853a-f5d5247a5573-kube-api-access-g97wl\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:35:36.149393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.149150 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:35:36.149393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.149161 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:35:36.149393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.149169 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:35:36.149393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.149176 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a4615b1-ff3d-4339-853a-f5d5247a5573-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:35:36.149393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.149196 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:35:36.149393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.149205 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a4615b1-ff3d-4339-853a-f5d5247a5573-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:35:36.149393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.149214 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4615b1-ff3d-4339-853a-f5d5247a5573-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:35:36.803344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.803314 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps" Oct 29 23:35:36.899706 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.899670 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps"] Oct 29 23:35:36.910507 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:36.910472 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-pfwps"] Oct 29 23:35:37.485732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:37.485696 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a4615b1-ff3d-4339-853a-f5d5247a5573" path="/var/lib/kubelet/pods/9a4615b1-ff3d-4339-853a-f5d5247a5573/volumes" Oct 29 23:35:38.486761 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:35:38.486728 5057 scope.go:122] "RemoveContainer" containerID="0957fa92c5d38be4195b35dc646131ab9e551749a298ff80df8f67070b262e71" Oct 29 23:36:04.021035 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:36:04.020995 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:36:11.740656 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:11.740612 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6"] Oct 29 23:36:11.742218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:11.742165 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9a4615b1-ff3d-4339-853a-f5d5247a5573" containerName="istio-proxy" Oct 29 23:36:11.742218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:11.742204 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a4615b1-ff3d-4339-853a-f5d5247a5573" containerName="istio-proxy" Oct 29 23:36:11.742454 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:11.742440 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a4615b1-ff3d-4339-853a-f5d5247a5573" containerName="istio-proxy" Oct 29 23:36:11.747711 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:11.747685 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" Oct 29 23:36:11.767358 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:11.767326 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b7a567-serving-cert\"" Oct 29 23:36:11.786208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:11.771893 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6"] Oct 29 23:36:11.891567 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:11.891524 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-metrics-certs\") pod \"maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6\" (UID: \"ba55c571-e46b-4ed5-af12-0cba3e7b44d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" Oct 29 23:36:11.891768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:11.891577 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqwms\" (UniqueName: \"kubernetes.io/projected/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-kube-api-access-hqwms\") pod \"maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6\" (UID: \"ba55c571-e46b-4ed5-af12-0cba3e7b44d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" Oct 29 23:36:11.891823 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:11.891781 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-maas-api-tls\") pod \"maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6\" (UID: \"ba55c571-e46b-4ed5-af12-0cba3e7b44d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" Oct 29 23:36:11.891992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:11.891967 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6\" (UID: \"ba55c571-e46b-4ed5-af12-0cba3e7b44d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" Oct 29 23:36:11.999305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:11.996253 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-metrics-certs\") pod \"maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6\" (UID: \"ba55c571-e46b-4ed5-af12-0cba3e7b44d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" Oct 29 23:36:12.024649 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:12.023324 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-metrics-certs\") pod \"maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6\" (UID: \"ba55c571-e46b-4ed5-af12-0cba3e7b44d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" Oct 29 23:36:12.024649 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:12.023922 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqwms\" (UniqueName: \"kubernetes.io/projected/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-kube-api-access-hqwms\") pod \"maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6\" (UID: \"ba55c571-e46b-4ed5-af12-0cba3e7b44d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" Oct 29 23:36:12.024649 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:12.024093 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-maas-api-tls\") pod \"maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6\" (UID: \"ba55c571-e46b-4ed5-af12-0cba3e7b44d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" Oct 29 23:36:12.024649 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:12.024243 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6\" (UID: \"ba55c571-e46b-4ed5-af12-0cba3e7b44d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" Oct 29 23:36:12.040207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:12.025154 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6\" (UID: \"ba55c571-e46b-4ed5-af12-0cba3e7b44d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" Oct 29 23:36:12.040207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:12.026673 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqwms\" (UniqueName: \"kubernetes.io/projected/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-kube-api-access-hqwms\") pod \"maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6\" (UID: \"ba55c571-e46b-4ed5-af12-0cba3e7b44d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" Oct 29 23:36:12.040207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:12.039921 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-maas-api-tls\") pod \"maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6\" (UID: \"ba55c571-e46b-4ed5-af12-0cba3e7b44d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" Oct 29 23:36:12.075224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:12.074967 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" Oct 29 23:36:12.409617 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:36:12.407867 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba55c571_e46b_4ed5_af12_0cba3e7b44d7.slice/crio-2b4ca57e36c4dd7c00dada2fdd65ca0e22bc7ca4bbd765911edd031b60e69509 WatchSource:0}: Error finding container 2b4ca57e36c4dd7c00dada2fdd65ca0e22bc7ca4bbd765911edd031b60e69509: Status 404 returned error can't find the container with id 2b4ca57e36c4dd7c00dada2fdd65ca0e22bc7ca4bbd765911edd031b60e69509 Oct 29 23:36:12.409617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:12.408933 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6"] Oct 29 23:36:12.409886 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:12.409864 5057 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:36:13.263981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:13.263926 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" event={"ID":"ba55c571-e46b-4ed5-af12-0cba3e7b44d7","Type":"ContainerStarted","Data":"c44b5a118d8df86c648a976f3f3ac7c05bf172ac4e83dee47eb55b05e6bfe382"} Oct 29 23:36:13.263981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:13.263987 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" event={"ID":"ba55c571-e46b-4ed5-af12-0cba3e7b44d7","Type":"ContainerStarted","Data":"2b4ca57e36c4dd7c00dada2fdd65ca0e22bc7ca4bbd765911edd031b60e69509"} Oct 29 23:36:13.265299 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:13.265238 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" Oct 29 23:36:13.312305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:13.312245 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" podStartSLOduration=1.95348732 podStartE2EDuration="2.312229185s" totalImagesPullingTime="358.741865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:36:13.309089666 +0000 UTC m=+3219.960588195" watchObservedRunningTime="2026-10-29 23:36:13.312229185 +0000 UTC m=+3219.963727715" Oct 29 23:36:15.246683 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:15.243839 5057 patch_prober.go:29] interesting pod/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": read tcp 10.132.0.2:37146->10.132.1.35:15021: read: connection reset by peer" start-of-body= Oct 29 23:36:15.246683 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:15.243896 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" podUID="03fd8f1b-0108-4471-a8bd-e1b98dbe8873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": read tcp 10.132.0.2:37146->10.132.1.35:15021: read: connection reset by peer" Oct 29 23:36:15.284006 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:15.283975 5057 generic.go:356] "Generic (PLEG): container finished" podID="03fd8f1b-0108-4471-a8bd-e1b98dbe8873" containerID="0106c8f7f7cfe71fdfb41aa0e9bc475858acd8cde0c57ecc59c810a0ca037e2f" exitCode=0 Oct 29 23:36:15.284148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:15.284054 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" event={"ID":"03fd8f1b-0108-4471-a8bd-e1b98dbe8873","Type":"ContainerDied","Data":"0106c8f7f7cfe71fdfb41aa0e9bc475858acd8cde0c57ecc59c810a0ca037e2f"} Oct 29 23:36:15.284691 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:15.284674 5057 scope.go:122] "RemoveContainer" containerID="0106c8f7f7cfe71fdfb41aa0e9bc475858acd8cde0c57ecc59c810a0ca037e2f" Oct 29 23:36:15.287034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:15.287005 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:36:15.296324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:15.287696 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:36:15.296324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:15.287843 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:36:16.295144 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:16.295106 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" event={"ID":"03fd8f1b-0108-4471-a8bd-e1b98dbe8873","Type":"ContainerStarted","Data":"1aa477f25ae102d5ef50530e01ea8784b1be272256c72f7876df960050258c84"} Oct 29 23:36:16.958546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:16.958509 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:36:16.958891 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:16.958869 5057 patch_prober.go:29] interesting pod/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 29 23:36:16.958950 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:16.958919 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" podUID="03fd8f1b-0108-4471-a8bd-e1b98dbe8873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 29 23:36:17.958961 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:17.958918 5057 patch_prober.go:29] interesting pod/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 29 23:36:17.959435 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:17.958988 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" podUID="03fd8f1b-0108-4471-a8bd-e1b98dbe8873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 29 23:36:18.958405 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:18.958365 5057 patch_prober.go:29] interesting pod/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 29 23:36:18.958652 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:18.958427 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" podUID="03fd8f1b-0108-4471-a8bd-e1b98dbe8873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 29 23:36:19.958914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:19.958871 5057 patch_prober.go:29] interesting pod/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 29 23:36:19.959365 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:19.958952 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" podUID="03fd8f1b-0108-4471-a8bd-e1b98dbe8873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 29 23:36:20.302650 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:20.302560 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" Oct 29 23:36:20.959682 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:20.959637 5057 patch_prober.go:29] interesting pod/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 29 23:36:20.960257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:20.959709 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" podUID="03fd8f1b-0108-4471-a8bd-e1b98dbe8873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 29 23:36:21.963926 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:21.963894 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:36:21.964507 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:21.964227 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:36:21.965224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:21.965164 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:36:33.915618 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:36:33.915582 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:36:42.263618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:42.263582 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:36:48.258917 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.258881 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt"] Oct 29 23:36:48.266612 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.266576 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:48.293199 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.293154 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt"] Oct 29 23:36:48.298023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.297997 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5cd0f2\"/\"openshift-service-ca.crt\"" Oct 29 23:36:48.298227 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.298006 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5cd0f2\"/\"kube-root-ca.crt\"" Oct 29 23:36:48.298227 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.298007 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5cd0f2\"/\"test-model-5cd0f2-kserve-self-signed-certs\"" Oct 29 23:36:48.298450 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.298104 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5cd0f2\"/\"default-dockercfg-vkqh9\"" Oct 29 23:36:48.331358 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.331317 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpxbn\" (UniqueName: \"kubernetes.io/projected/9033febc-c7ee-4884-9145-c27229f9f5dd-kube-api-access-kpxbn\") pod \"test-model-5cd0f2-kserve-5646c5b5c9-dx6pt\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:48.331561 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.331521 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-dshm\") pod \"test-model-5cd0f2-kserve-5646c5b5c9-dx6pt\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:48.331644 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.331585 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-home\") pod \"test-model-5cd0f2-kserve-5646c5b5c9-dx6pt\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:48.331709 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.331648 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-model-cache\") pod \"test-model-5cd0f2-kserve-5646c5b5c9-dx6pt\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:48.331776 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.331744 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9033febc-c7ee-4884-9145-c27229f9f5dd-tls-certs\") pod \"test-model-5cd0f2-kserve-5646c5b5c9-dx6pt\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:48.331855 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.331839 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-tmp-dir\") pod \"test-model-5cd0f2-kserve-5646c5b5c9-dx6pt\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:48.434955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.434904 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-home\") pod \"test-model-5cd0f2-kserve-5646c5b5c9-dx6pt\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:48.435291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.435257 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-home\") pod \"test-model-5cd0f2-kserve-5646c5b5c9-dx6pt\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:48.435397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.435375 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-model-cache\") pod \"test-model-5cd0f2-kserve-5646c5b5c9-dx6pt\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:48.435562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.435532 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9033febc-c7ee-4884-9145-c27229f9f5dd-tls-certs\") pod \"test-model-5cd0f2-kserve-5646c5b5c9-dx6pt\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:48.435662 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.435642 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-tmp-dir\") pod \"test-model-5cd0f2-kserve-5646c5b5c9-dx6pt\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:48.435754 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.435701 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-model-cache\") pod \"test-model-5cd0f2-kserve-5646c5b5c9-dx6pt\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:48.435894 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.435873 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpxbn\" (UniqueName: \"kubernetes.io/projected/9033febc-c7ee-4884-9145-c27229f9f5dd-kube-api-access-kpxbn\") pod \"test-model-5cd0f2-kserve-5646c5b5c9-dx6pt\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:48.436036 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.436005 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-dshm\") pod \"test-model-5cd0f2-kserve-5646c5b5c9-dx6pt\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:48.436036 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.436024 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-tmp-dir\") pod \"test-model-5cd0f2-kserve-5646c5b5c9-dx6pt\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:48.438611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.438561 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-dshm\") pod \"test-model-5cd0f2-kserve-5646c5b5c9-dx6pt\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:48.439728 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.439707 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9033febc-c7ee-4884-9145-c27229f9f5dd-tls-certs\") pod \"test-model-5cd0f2-kserve-5646c5b5c9-dx6pt\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:48.467780 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.467749 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpxbn\" (UniqueName: \"kubernetes.io/projected/9033febc-c7ee-4884-9145-c27229f9f5dd-kube-api-access-kpxbn\") pod \"test-model-5cd0f2-kserve-5646c5b5c9-dx6pt\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:48.598689 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:48.598602 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:49.160895 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:36:49.160863 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9033febc_c7ee_4884_9145_c27229f9f5dd.slice/crio-e9db943eb609dc71c35cdc0d476f7bf22719119d9e36758f12f1cdb900b2300a WatchSource:0}: Error finding container e9db943eb609dc71c35cdc0d476f7bf22719119d9e36758f12f1cdb900b2300a: Status 404 returned error can't find the container with id e9db943eb609dc71c35cdc0d476f7bf22719119d9e36758f12f1cdb900b2300a Oct 29 23:36:49.163977 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:49.163952 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt"] Oct 29 23:36:49.877772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:49.877678 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" event={"ID":"9033febc-c7ee-4884-9145-c27229f9f5dd","Type":"ContainerStarted","Data":"c8c02cc5919139e1d257fc16fb1d7cf79f573f42f43d57d1a2ad258ed94f0b8e"} Oct 29 23:36:49.877772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:49.877730 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" event={"ID":"9033febc-c7ee-4884-9145-c27229f9f5dd","Type":"ContainerStarted","Data":"e9db943eb609dc71c35cdc0d476f7bf22719119d9e36758f12f1cdb900b2300a"} Oct 29 23:36:49.923420 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:49.923360 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" podStartSLOduration=1.9233437389999999 podStartE2EDuration="1.923343739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:36:49.91860114 +0000 UTC m=+3256.570099669" watchObservedRunningTime="2026-10-29 23:36:49.923343739 +0000 UTC m=+3256.574842271" Oct 29 23:36:50.600088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:50.600050 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:58.600147 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:58.600108 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:58.613599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:58.613554 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:36:58.982875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:36:58.982840 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:37:03.915996 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:03.915952 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:37:22.501231 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:22.500874 5057 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e1769a/agc-cred-757501a65e29: secret "agc-cred-757501a65e29" not found Oct 29 23:37:22.501231 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:22.500907 5057 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b: secret "agc-cred-757501a65e29" not found Oct 29 23:37:22.501231 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:22.500974 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-provider-credentials podName:2aa921cc-cfbe-4194-9ca4-91c9f999cf1a nodeName:}" failed. No retries permitted until 2026-10-29 23:37:23.000951779 +0000 UTC m=+3289.652450310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" (UID: "2aa921cc-cfbe-4194-9ca4-91c9f999cf1a") : secret "agc-cred-757501a65e29" not found Oct 29 23:37:23.011556 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:23.011522 5057 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e1769a/agc-cred-757501a65e29: secret "agc-cred-757501a65e29" not found Oct 29 23:37:23.011556 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:23.011551 5057 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b: secret "agc-cred-757501a65e29" not found Oct 29 23:37:23.011779 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:23.011606 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-provider-credentials podName:2aa921cc-cfbe-4194-9ca4-91c9f999cf1a nodeName:}" failed. No retries permitted until 2026-10-29 23:37:24.011591507 +0000 UTC m=+3290.663090014 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" (UID: "2aa921cc-cfbe-4194-9ca4-91c9f999cf1a") : secret "agc-cred-757501a65e29" not found Oct 29 23:37:24.047508 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:24.047474 5057 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e1769a/agc-cred-757501a65e29: secret "agc-cred-757501a65e29" not found Oct 29 23:37:24.047508 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:24.047500 5057 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b: secret "agc-cred-757501a65e29" not found Oct 29 23:37:24.047944 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:24.047562 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-provider-credentials podName:2aa921cc-cfbe-4194-9ca4-91c9f999cf1a nodeName:}" failed. No retries permitted until 2026-10-29 23:37:26.047547353 +0000 UTC m=+3292.699045860 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" (UID: "2aa921cc-cfbe-4194-9ca4-91c9f999cf1a") : secret "agc-cred-757501a65e29" not found Oct 29 23:37:25.295171 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:25.295138 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d"] Oct 29 23:37:26.116770 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:26.116732 5057 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e1769a/agc-cred-757501a65e29: secret "agc-cred-757501a65e29" not found Oct 29 23:37:26.116770 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:26.116759 5057 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b: secret "agc-cred-757501a65e29" not found Oct 29 23:37:26.117051 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:26.116823 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-provider-credentials podName:2aa921cc-cfbe-4194-9ca4-91c9f999cf1a nodeName:}" failed. No retries permitted until 2026-10-29 23:37:30.116808315 +0000 UTC m=+3296.768306823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" (UID: "2aa921cc-cfbe-4194-9ca4-91c9f999cf1a") : secret "agc-cred-757501a65e29" not found Oct 29 23:37:30.129351 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:30.129315 5057 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e1769a/agc-cred-757501a65e29: secret "agc-cred-757501a65e29" not found Oct 29 23:37:30.129351 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:30.129343 5057 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b: secret "agc-cred-757501a65e29" not found Oct 29 23:37:30.129809 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:30.129430 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-provider-credentials podName:2aa921cc-cfbe-4194-9ca4-91c9f999cf1a nodeName:}" failed. No retries permitted until 2026-10-29 23:37:38.129413417 +0000 UTC m=+3304.780911930 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" (UID: "2aa921cc-cfbe-4194-9ca4-91c9f999cf1a") : secret "agc-cred-757501a65e29" not found Oct 29 23:37:30.295596 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.295555 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" podUID="6654e558-e0af-412c-94a3-5c958c854a4e" containerName="payload-processing" containerID="cri-o://8cc49af359974d3aa10aac026dac2e52c6182ecddbd6077b80b6c79ed29e2eee" gracePeriod=25 Oct 29 23:37:30.619957 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.619928 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" Oct 29 23:37:30.638883 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.638847 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6654e558-e0af-412c-94a3-5c958c854a4e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6654e558-e0af-412c-94a3-5c958c854a4e-provider-credentials\") pod \"6654e558-e0af-412c-94a3-5c958c854a4e\" (UID: \"6654e558-e0af-412c-94a3-5c958c854a4e\") " Oct 29 23:37:30.639027 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.638891 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6654e558-e0af-412c-94a3-5c958c854a4e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6654e558-e0af-412c-94a3-5c958c854a4e-routing-overlay-volume\") pod \"6654e558-e0af-412c-94a3-5c958c854a4e\" (UID: \"6654e558-e0af-412c-94a3-5c958c854a4e\") " Oct 29 23:37:30.639093 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.639064 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6654e558-e0af-412c-94a3-5c958c854a4e-kube-api-access-tl68b\" (UniqueName: \"kubernetes.io/projected/6654e558-e0af-412c-94a3-5c958c854a4e-kube-api-access-tl68b\") pod \"6654e558-e0af-412c-94a3-5c958c854a4e\" (UID: \"6654e558-e0af-412c-94a3-5c958c854a4e\") " Oct 29 23:37:30.639399 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.639371 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6654e558-e0af-412c-94a3-5c958c854a4e-routing-overlay-volume" pod "6654e558-e0af-412c-94a3-5c958c854a4e" (UID: "6654e558-e0af-412c-94a3-5c958c854a4e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:30.639531 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.639507 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6654e558-e0af-412c-94a3-5c958c854a4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6654e558-e0af-412c-94a3-5c958c854a4e-plugins-config-volume\") pod \"6654e558-e0af-412c-94a3-5c958c854a4e\" (UID: \"6654e558-e0af-412c-94a3-5c958c854a4e\") " Oct 29 23:37:30.639988 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.639963 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6654e558-e0af-412c-94a3-5c958c854a4e-plugins-config-volume" pod "6654e558-e0af-412c-94a3-5c958c854a4e" (UID: "6654e558-e0af-412c-94a3-5c958c854a4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:30.640271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.640254 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6654e558-e0af-412c-94a3-5c958c854a4e-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:37:30.640340 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.640272 5057 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6654e558-e0af-412c-94a3-5c958c854a4e-routing-overlay-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:37:30.641324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.641298 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6654e558-e0af-412c-94a3-5c958c854a4e-provider-credentials" pod "6654e558-e0af-412c-94a3-5c958c854a4e" (UID: "6654e558-e0af-412c-94a3-5c958c854a4e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:30.641410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.641333 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6654e558-e0af-412c-94a3-5c958c854a4e-kube-api-access-tl68b" pod "6654e558-e0af-412c-94a3-5c958c854a4e" (UID: "6654e558-e0af-412c-94a3-5c958c854a4e"). InnerVolumeSpecName "kube-api-access-tl68b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:30.742538 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.742496 5057 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6654e558-e0af-412c-94a3-5c958c854a4e-provider-credentials\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:37:30.742538 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.742532 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl68b\" (UniqueName: \"kubernetes.io/projected/6654e558-e0af-412c-94a3-5c958c854a4e-kube-api-access-tl68b\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:37:30.850725 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.850680 5057 generic.go:356] "Generic (PLEG): container finished" podID="6654e558-e0af-412c-94a3-5c958c854a4e" containerID="8cc49af359974d3aa10aac026dac2e52c6182ecddbd6077b80b6c79ed29e2eee" exitCode=0 Oct 29 23:37:30.851008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.850749 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" event={"ID":"6654e558-e0af-412c-94a3-5c958c854a4e","Type":"ContainerDied","Data":"8cc49af359974d3aa10aac026dac2e52c6182ecddbd6077b80b6c79ed29e2eee"} Oct 29 23:37:30.851008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.850823 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" event={"ID":"6654e558-e0af-412c-94a3-5c958c854a4e","Type":"ContainerDied","Data":"d194cb9949a626637282b8f836b744d342cd5a8e31b5cc1e260e7ca4a4c16b25"} Oct 29 23:37:30.851008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.850751 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d" Oct 29 23:37:30.851008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.850849 5057 scope.go:122] "RemoveContainer" containerID="8cc49af359974d3aa10aac026dac2e52c6182ecddbd6077b80b6c79ed29e2eee" Oct 29 23:37:30.872718 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.872694 5057 scope.go:122] "RemoveContainer" containerID="8cc49af359974d3aa10aac026dac2e52c6182ecddbd6077b80b6c79ed29e2eee" Oct 29 23:37:30.873050 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:30.873025 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc49af359974d3aa10aac026dac2e52c6182ecddbd6077b80b6c79ed29e2eee\": container with ID starting with 8cc49af359974d3aa10aac026dac2e52c6182ecddbd6077b80b6c79ed29e2eee not found: ID does not exist" containerID="8cc49af359974d3aa10aac026dac2e52c6182ecddbd6077b80b6c79ed29e2eee" Oct 29 23:37:30.873101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.873063 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc49af359974d3aa10aac026dac2e52c6182ecddbd6077b80b6c79ed29e2eee"} err="failed to get container status \"8cc49af359974d3aa10aac026dac2e52c6182ecddbd6077b80b6c79ed29e2eee\": rpc error: code = NotFound desc = could not find container \"8cc49af359974d3aa10aac026dac2e52c6182ecddbd6077b80b6c79ed29e2eee\": container with ID starting with 8cc49af359974d3aa10aac026dac2e52c6182ecddbd6077b80b6c79ed29e2eee not found: ID does not exist" Oct 29 23:37:30.913329 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.913263 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d"] Oct 29 23:37:30.921064 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:30.921034 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-55b6gmt7d"] Oct 29 23:37:31.488494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:31.488464 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6654e558-e0af-412c-94a3-5c958c854a4e" path="/var/lib/kubelet/pods/6654e558-e0af-412c-94a3-5c958c854a4e/volumes" Oct 29 23:37:31.558335 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:31.558296 5057 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-206095/agc-cred-873d12f62d54: secret "agc-cred-873d12f62d54" not found Oct 29 23:37:31.558335 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:31.558325 5057 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l: secret "agc-cred-873d12f62d54" not found Oct 29 23:37:31.558640 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:31.558381 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-provider-credentials podName:0a1f5545-df3e-4bdf-b69d-dd4ef38d5072 nodeName:}" failed. No retries permitted until 2026-10-29 23:37:32.058361692 +0000 UTC m=+3298.709860228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" (UID: "0a1f5545-df3e-4bdf-b69d-dd4ef38d5072") : secret "agc-cred-873d12f62d54" not found Oct 29 23:37:32.077125 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:32.077089 5057 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-206095/agc-cred-873d12f62d54: secret "agc-cred-873d12f62d54" not found Oct 29 23:37:32.077387 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:32.077367 5057 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l: secret "agc-cred-873d12f62d54" not found Oct 29 23:37:32.077709 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:32.077689 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-provider-credentials podName:0a1f5545-df3e-4bdf-b69d-dd4ef38d5072 nodeName:}" failed. No retries permitted until 2026-10-29 23:37:33.077668453 +0000 UTC m=+3299.729166977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" (UID: "0a1f5545-df3e-4bdf-b69d-dd4ef38d5072") : secret "agc-cred-873d12f62d54" not found Oct 29 23:37:33.096457 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:33.096429 5057 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-206095/agc-cred-873d12f62d54: secret "agc-cred-873d12f62d54" not found Oct 29 23:37:33.096457 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:33.096452 5057 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l: secret "agc-cred-873d12f62d54" not found Oct 29 23:37:33.096970 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:33.096508 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-provider-credentials podName:0a1f5545-df3e-4bdf-b69d-dd4ef38d5072 nodeName:}" failed. No retries permitted until 2026-10-29 23:37:35.09649159 +0000 UTC m=+3301.747990112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" (UID: "0a1f5545-df3e-4bdf-b69d-dd4ef38d5072") : secret "agc-cred-873d12f62d54" not found Oct 29 23:37:33.951915 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:33.951877 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:37:34.120931 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:34.120896 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv"] Oct 29 23:37:34.122294 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:34.122270 5057 cpu_manager.go:414] "containerMap: removing container" podUID="6654e558-e0af-412c-94a3-5c958c854a4e" containerName="payload-processing" Oct 29 23:37:34.122294 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:34.122294 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6654e558-e0af-412c-94a3-5c958c854a4e" containerName="payload-processing" Oct 29 23:37:34.122607 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:34.122593 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="6654e558-e0af-412c-94a3-5c958c854a4e" containerName="payload-processing" Oct 29 23:37:34.128918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:34.128894 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv" Oct 29 23:37:34.152259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:34.152230 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv"] Oct 29 23:37:34.154779 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:34.154732 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a8ab3c\"" Oct 29 23:37:34.154913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:34.154851 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a8ab3c-dockercfg-2f9rp\"" Oct 29 23:37:34.251947 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:34.251627 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664aecb9-c880-4cb4-806a-293858b56120-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv\" (UID: \"664aecb9-c880-4cb4-806a-293858b56120\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv" Oct 29 23:37:34.251947 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:34.251725 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hms5\" (UniqueName: \"kubernetes.io/projected/664aecb9-c880-4cb4-806a-293858b56120-kube-api-access-8hms5\") pod \"payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv\" (UID: \"664aecb9-c880-4cb4-806a-293858b56120\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv" Oct 29 23:37:34.366209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:34.354922 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664aecb9-c880-4cb4-806a-293858b56120-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv\" (UID: \"664aecb9-c880-4cb4-806a-293858b56120\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv" Oct 29 23:37:34.366608 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:34.366572 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hms5\" (UniqueName: \"kubernetes.io/projected/664aecb9-c880-4cb4-806a-293858b56120-kube-api-access-8hms5\") pod \"payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv\" (UID: \"664aecb9-c880-4cb4-806a-293858b56120\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv" Oct 29 23:37:34.381209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:34.368297 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664aecb9-c880-4cb4-806a-293858b56120-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv\" (UID: \"664aecb9-c880-4cb4-806a-293858b56120\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv" Oct 29 23:37:34.695754 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:34.651900 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hms5\" (UniqueName: \"kubernetes.io/projected/664aecb9-c880-4cb4-806a-293858b56120-kube-api-access-8hms5\") pod \"payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv\" (UID: \"664aecb9-c880-4cb4-806a-293858b56120\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv" Oct 29 23:37:34.759527 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:34.759485 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a8ab3c-dockercfg-2f9rp\"" Oct 29 23:37:34.759956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:34.759938 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv" Oct 29 23:37:34.885824 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:34.873932 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l"] Oct 29 23:37:35.103079 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:37:35.103054 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664aecb9_c880_4cb4_806a_293858b56120.slice/crio-8d3af56d863544b32f25bc5ad8cd4410b33355e6c546f6c5f65200848bf5e11b WatchSource:0}: Error finding container 8d3af56d863544b32f25bc5ad8cd4410b33355e6c546f6c5f65200848bf5e11b: Status 404 returned error can't find the container with id 8d3af56d863544b32f25bc5ad8cd4410b33355e6c546f6c5f65200848bf5e11b Oct 29 23:37:35.103429 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:35.103407 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv"] Oct 29 23:37:35.151431 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:35.151401 5057 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-206095/agc-cred-873d12f62d54: secret "agc-cred-873d12f62d54" not found Oct 29 23:37:35.151431 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:35.151429 5057 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l: secret "agc-cred-873d12f62d54" not found Oct 29 23:37:35.151945 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:35.151520 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-provider-credentials podName:0a1f5545-df3e-4bdf-b69d-dd4ef38d5072 nodeName:}" failed. No retries permitted until 2026-10-29 23:37:39.151486766 +0000 UTC m=+3305.802985289 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" (UID: "0a1f5545-df3e-4bdf-b69d-dd4ef38d5072") : secret "agc-cred-873d12f62d54" not found Oct 29 23:37:35.190012 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:35.189985 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:37:35.202584 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:35.202548 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:37:35.496776 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:35.496742 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b"] Oct 29 23:37:35.928621 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:35.928583 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv" event={"ID":"664aecb9-c880-4cb4-806a-293858b56120","Type":"ContainerStarted","Data":"8da79cdf2cf1e6cba6707f76965a0da907589df30229ba43694930b11555e70c"} Oct 29 23:37:35.928794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:35.928643 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv" event={"ID":"664aecb9-c880-4cb4-806a-293858b56120","Type":"ContainerStarted","Data":"8d3af56d863544b32f25bc5ad8cd4410b33355e6c546f6c5f65200848bf5e11b"} Oct 29 23:37:35.928794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:35.928671 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv" Oct 29 23:37:35.977012 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:35.976957 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv" podStartSLOduration=1.976943203 podStartE2EDuration="1.976943203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:37:35.973199872 +0000 UTC m=+3302.624698403" watchObservedRunningTime="2026-10-29 23:37:35.976943203 +0000 UTC m=+3302.628441732" Oct 29 23:37:38.172637 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:38.172599 5057 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e1769a/agc-cred-757501a65e29: secret "agc-cred-757501a65e29" not found Oct 29 23:37:38.172637 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:38.172631 5057 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b: secret "agc-cred-757501a65e29" not found Oct 29 23:37:38.173246 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:38.172700 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-provider-credentials podName:2aa921cc-cfbe-4194-9ca4-91c9f999cf1a nodeName:}" failed. No retries permitted until 2026-10-29 23:37:54.172680838 +0000 UTC m=+3320.824179349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" (UID: "2aa921cc-cfbe-4194-9ca4-91c9f999cf1a") : secret "agc-cred-757501a65e29" not found Oct 29 23:37:39.246843 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:39.246799 5057 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-206095/agc-cred-873d12f62d54: secret "agc-cred-873d12f62d54" not found Oct 29 23:37:39.247446 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:39.247426 5057 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l: secret "agc-cred-873d12f62d54" not found Oct 29 23:37:39.247644 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:39.247631 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-provider-credentials podName:0a1f5545-df3e-4bdf-b69d-dd4ef38d5072 nodeName:}" failed. No retries permitted until 2026-10-29 23:37:47.247603139 +0000 UTC m=+3313.899101664 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" (UID: "0a1f5545-df3e-4bdf-b69d-dd4ef38d5072") : secret "agc-cred-873d12f62d54" not found Oct 29 23:37:39.290169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:39.288326 5057 generic.go:356] "Generic (PLEG): container finished" podID="55c4bb94-26cc-45e9-ac4c-3542b1758fcb" containerID="821940f367cba577a59738e0d12b4f298326901950decec0b4654b3a0423689a" exitCode=137 Oct 29 23:37:39.290169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:39.288649 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt" event={"ID":"55c4bb94-26cc-45e9-ac4c-3542b1758fcb","Type":"ContainerDied","Data":"821940f367cba577a59738e0d12b4f298326901950decec0b4654b3a0423689a"} Oct 29 23:37:39.290169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:39.289102 5057 scope.go:122] "RemoveContainer" containerID="821940f367cba577a59738e0d12b4f298326901950decec0b4654b3a0423689a" Oct 29 23:37:39.709216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:39.702290 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt" Oct 29 23:37:39.875421 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:39.875381 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" podUID="0a1f5545-df3e-4bdf-b69d-dd4ef38d5072" containerName="payload-processing" containerID="cri-o://dd08d5fcd850e163059930ee7b08b05e041a226d1a7043737d9ba87e5d94eba7" gracePeriod=25 Oct 29 23:37:40.176834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.176805 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" Oct 29 23:37:40.275453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.275380 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-plugins-config-volume\") pod \"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072\" (UID: \"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072\") " Oct 29 23:37:40.275453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.275449 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-provider-credentials\") pod \"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072\" (UID: \"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072\") " Oct 29 23:37:40.275981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.275586 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-routing-overlay-volume\") pod \"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072\" (UID: \"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072\") " Oct 29 23:37:40.275981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.275917 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-plugins-config-volume" pod "0a1f5545-df3e-4bdf-b69d-dd4ef38d5072" (UID: "0a1f5545-df3e-4bdf-b69d-dd4ef38d5072"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:40.275981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.275936 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-kube-api-access-b8jq7\" (UniqueName: \"kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-kube-api-access-b8jq7\") pod \"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072\" (UID: \"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072\") " Oct 29 23:37:40.276200 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.275982 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-routing-overlay-volume" pod "0a1f5545-df3e-4bdf-b69d-dd4ef38d5072" (UID: "0a1f5545-df3e-4bdf-b69d-dd4ef38d5072"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:40.276759 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.276738 5057 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-routing-overlay-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:37:40.276823 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.276761 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:37:40.277749 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.277726 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-provider-credentials" pod "0a1f5545-df3e-4bdf-b69d-dd4ef38d5072" (UID: "0a1f5545-df3e-4bdf-b69d-dd4ef38d5072"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:40.277920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.277904 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-kube-api-access-b8jq7" pod "0a1f5545-df3e-4bdf-b69d-dd4ef38d5072" (UID: "0a1f5545-df3e-4bdf-b69d-dd4ef38d5072"). InnerVolumeSpecName "kube-api-access-b8jq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:40.299298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.299265 5057 generic.go:356] "Generic (PLEG): container finished" podID="0a1f5545-df3e-4bdf-b69d-dd4ef38d5072" containerID="dd08d5fcd850e163059930ee7b08b05e041a226d1a7043737d9ba87e5d94eba7" exitCode=0 Oct 29 23:37:40.299458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.299316 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" Oct 29 23:37:40.299458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.299348 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" event={"ID":"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072","Type":"ContainerDied","Data":"dd08d5fcd850e163059930ee7b08b05e041a226d1a7043737d9ba87e5d94eba7"} Oct 29 23:37:40.299458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.299396 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l" event={"ID":"0a1f5545-df3e-4bdf-b69d-dd4ef38d5072","Type":"ContainerDied","Data":"f589eab0c5df0835e172f2b0eeb0c5eb46a66e70cea90f7aa0ddab5f589679b2"} Oct 29 23:37:40.299458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.299418 5057 scope.go:122] "RemoveContainer" containerID="dd08d5fcd850e163059930ee7b08b05e041a226d1a7043737d9ba87e5d94eba7" Oct 29 23:37:40.302078 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.302053 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt" event={"ID":"55c4bb94-26cc-45e9-ac4c-3542b1758fcb","Type":"ContainerStarted","Data":"86b1f5e6ef3b21bdad4e9a877fd8ca6599f34f8eed241156af679cd521ae3422"} Oct 29 23:37:40.302245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.302121 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt" Oct 29 23:37:40.310987 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.310967 5057 scope.go:122] "RemoveContainer" containerID="dd08d5fcd850e163059930ee7b08b05e041a226d1a7043737d9ba87e5d94eba7" Oct 29 23:37:40.311291 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:40.311253 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd08d5fcd850e163059930ee7b08b05e041a226d1a7043737d9ba87e5d94eba7\": container with ID starting with dd08d5fcd850e163059930ee7b08b05e041a226d1a7043737d9ba87e5d94eba7 not found: ID does not exist" containerID="dd08d5fcd850e163059930ee7b08b05e041a226d1a7043737d9ba87e5d94eba7" Oct 29 23:37:40.311371 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.311306 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd08d5fcd850e163059930ee7b08b05e041a226d1a7043737d9ba87e5d94eba7"} err="failed to get container status \"dd08d5fcd850e163059930ee7b08b05e041a226d1a7043737d9ba87e5d94eba7\": rpc error: code = NotFound desc = could not find container \"dd08d5fcd850e163059930ee7b08b05e041a226d1a7043737d9ba87e5d94eba7\": container with ID starting with dd08d5fcd850e163059930ee7b08b05e041a226d1a7043737d9ba87e5d94eba7 not found: ID does not exist" Oct 29 23:37:40.378987 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.378951 5057 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-provider-credentials\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:37:40.378987 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.378986 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8jq7\" (UniqueName: \"kubernetes.io/projected/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072-kube-api-access-b8jq7\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:37:40.420077 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.420045 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l"] Oct 29 23:37:40.435298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.427703 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-9cf6jng5l"] Oct 29 23:37:40.497799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.497741 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" podUID="2aa921cc-cfbe-4194-9ca4-91c9f999cf1a" containerName="payload-processing" containerID="cri-o://45253183432f835ee53b56c8f94b908947028e547fe421da82f4040e291b0efc" gracePeriod=25 Oct 29 23:37:40.809045 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:40.809018 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" Oct 29 23:37:41.021216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.012325 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-routing-overlay-volume\") pod \"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a\" (UID: \"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a\") " Oct 29 23:37:41.021216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.012524 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-provider-credentials\") pod \"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a\" (UID: \"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a\") " Oct 29 23:37:41.021216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.012851 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-plugins-config-volume\") pod \"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a\" (UID: \"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a\") " Oct 29 23:37:41.021216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.012940 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-kube-api-access-nq7gs\" (UniqueName: \"kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-kube-api-access-nq7gs\") pod \"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a\" (UID: \"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a\") " Oct 29 23:37:41.032303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.029812 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-plugins-config-volume" pod "2aa921cc-cfbe-4194-9ca4-91c9f999cf1a" (UID: "2aa921cc-cfbe-4194-9ca4-91c9f999cf1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:41.032303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.030831 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-routing-overlay-volume" pod "2aa921cc-cfbe-4194-9ca4-91c9f999cf1a" (UID: "2aa921cc-cfbe-4194-9ca4-91c9f999cf1a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:41.078732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.076289 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-kube-api-access-nq7gs" pod "2aa921cc-cfbe-4194-9ca4-91c9f999cf1a" (UID: "2aa921cc-cfbe-4194-9ca4-91c9f999cf1a"). InnerVolumeSpecName "kube-api-access-nq7gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:41.078732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.076395 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-provider-credentials" pod "2aa921cc-cfbe-4194-9ca4-91c9f999cf1a" (UID: "2aa921cc-cfbe-4194-9ca4-91c9f999cf1a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:41.116226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.116178 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:37:41.116226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.116223 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq7gs\" (UniqueName: \"kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-kube-api-access-nq7gs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:37:41.116226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.116232 5057 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-routing-overlay-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:37:41.116467 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.116241 5057 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a-provider-credentials\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:37:41.311723 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.311641 5057 generic.go:356] "Generic (PLEG): container finished" podID="2aa921cc-cfbe-4194-9ca4-91c9f999cf1a" containerID="45253183432f835ee53b56c8f94b908947028e547fe421da82f4040e291b0efc" exitCode=0 Oct 29 23:37:41.311723 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.311688 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" Oct 29 23:37:41.312260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.311717 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" event={"ID":"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a","Type":"ContainerDied","Data":"45253183432f835ee53b56c8f94b908947028e547fe421da82f4040e291b0efc"} Oct 29 23:37:41.312260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.311767 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b" event={"ID":"2aa921cc-cfbe-4194-9ca4-91c9f999cf1a","Type":"ContainerDied","Data":"d94f25761fcaded38ff561e64ca793cee34203600d1c8d3625a517c6bcd2f140"} Oct 29 23:37:41.312260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.311789 5057 scope.go:122] "RemoveContainer" containerID="45253183432f835ee53b56c8f94b908947028e547fe421da82f4040e291b0efc" Oct 29 23:37:41.339240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.339218 5057 scope.go:122] "RemoveContainer" containerID="45253183432f835ee53b56c8f94b908947028e547fe421da82f4040e291b0efc" Oct 29 23:37:41.339556 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:37:41.339536 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45253183432f835ee53b56c8f94b908947028e547fe421da82f4040e291b0efc\": container with ID starting with 45253183432f835ee53b56c8f94b908947028e547fe421da82f4040e291b0efc not found: ID does not exist" containerID="45253183432f835ee53b56c8f94b908947028e547fe421da82f4040e291b0efc" Oct 29 23:37:41.339607 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.339565 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45253183432f835ee53b56c8f94b908947028e547fe421da82f4040e291b0efc"} err="failed to get container status \"45253183432f835ee53b56c8f94b908947028e547fe421da82f4040e291b0efc\": rpc error: code = NotFound desc = could not find container \"45253183432f835ee53b56c8f94b908947028e547fe421da82f4040e291b0efc\": container with ID starting with 45253183432f835ee53b56c8f94b908947028e547fe421da82f4040e291b0efc not found: ID does not exist" Oct 29 23:37:41.377523 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.377485 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b"] Oct 29 23:37:41.388602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.388575 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-75999zg5b"] Oct 29 23:37:41.485031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.484996 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a1f5545-df3e-4bdf-b69d-dd4ef38d5072" path="/var/lib/kubelet/pods/0a1f5545-df3e-4bdf-b69d-dd4ef38d5072/volumes" Oct 29 23:37:41.485538 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:41.485523 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aa921cc-cfbe-4194-9ca4-91c9f999cf1a" path="/var/lib/kubelet/pods/2aa921cc-cfbe-4194-9ca4-91c9f999cf1a/volumes" Oct 29 23:37:46.945671 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:46.945637 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv" Oct 29 23:37:47.615371 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.615337 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw"] Oct 29 23:37:47.616265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.616248 5057 cpu_manager.go:414] "containerMap: removing container" podUID="0a1f5545-df3e-4bdf-b69d-dd4ef38d5072" containerName="payload-processing" Oct 29 23:37:47.616265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.616265 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1f5545-df3e-4bdf-b69d-dd4ef38d5072" containerName="payload-processing" Oct 29 23:37:47.616354 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.616289 5057 cpu_manager.go:414] "containerMap: removing container" podUID="2aa921cc-cfbe-4194-9ca4-91c9f999cf1a" containerName="payload-processing" Oct 29 23:37:47.616354 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.616294 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa921cc-cfbe-4194-9ca4-91c9f999cf1a" containerName="payload-processing" Oct 29 23:37:47.616494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.616483 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1f5545-df3e-4bdf-b69d-dd4ef38d5072" containerName="payload-processing" Oct 29 23:37:47.616538 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.616497 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa921cc-cfbe-4194-9ca4-91c9f999cf1a" containerName="payload-processing" Oct 29 23:37:47.621979 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.621954 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw" Oct 29 23:37:47.639343 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.639321 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3b8149\"" Oct 29 23:37:47.641293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.641269 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw"] Oct 29 23:37:47.674482 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.674450 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b"] Oct 29 23:37:47.679875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.679853 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b" Oct 29 23:37:47.696000 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.695953 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3b8149-dockercfg-djfwd\"" Oct 29 23:37:47.702120 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.702098 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b"] Oct 29 23:37:47.731085 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.731052 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65brg\" (UniqueName: \"kubernetes.io/projected/ed4c81ea-a9b7-43c6-a893-342cd3d4389f-kube-api-access-65brg\") pod \"payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw\" (UID: \"ed4c81ea-a9b7-43c6-a893-342cd3d4389f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw" Oct 29 23:37:47.731297 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.731273 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4c81ea-a9b7-43c6-a893-342cd3d4389f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw\" (UID: \"ed4c81ea-a9b7-43c6-a893-342cd3d4389f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw" Oct 29 23:37:47.847223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.842889 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65brg\" (UniqueName: \"kubernetes.io/projected/ed4c81ea-a9b7-43c6-a893-342cd3d4389f-kube-api-access-65brg\") pod \"payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw\" (UID: \"ed4c81ea-a9b7-43c6-a893-342cd3d4389f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw" Oct 29 23:37:47.847223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.842953 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bp4x\" (UniqueName: \"kubernetes.io/projected/a8aa349b-e04a-4726-9b2e-a910d328e7f9-kube-api-access-4bp4x\") pod \"payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b\" (UID: \"a8aa349b-e04a-4726-9b2e-a910d328e7f9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b" Oct 29 23:37:47.847223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.843067 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4c81ea-a9b7-43c6-a893-342cd3d4389f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw\" (UID: \"ed4c81ea-a9b7-43c6-a893-342cd3d4389f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw" Oct 29 23:37:47.847223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.843114 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8aa349b-e04a-4726-9b2e-a910d328e7f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b\" (UID: \"a8aa349b-e04a-4726-9b2e-a910d328e7f9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b" Oct 29 23:37:47.851788 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.851757 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4c81ea-a9b7-43c6-a893-342cd3d4389f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw\" (UID: \"ed4c81ea-a9b7-43c6-a893-342cd3d4389f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw" Oct 29 23:37:47.936774 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.935526 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65brg\" (UniqueName: \"kubernetes.io/projected/ed4c81ea-a9b7-43c6-a893-342cd3d4389f-kube-api-access-65brg\") pod \"payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw\" (UID: \"ed4c81ea-a9b7-43c6-a893-342cd3d4389f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw" Oct 29 23:37:47.938543 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.938315 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw" Oct 29 23:37:47.959778 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.949796 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8aa349b-e04a-4726-9b2e-a910d328e7f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b\" (UID: \"a8aa349b-e04a-4726-9b2e-a910d328e7f9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b" Oct 29 23:37:47.962669 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.962566 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8aa349b-e04a-4726-9b2e-a910d328e7f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b\" (UID: \"a8aa349b-e04a-4726-9b2e-a910d328e7f9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b" Oct 29 23:37:47.977968 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:47.972386 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bp4x\" (UniqueName: \"kubernetes.io/projected/a8aa349b-e04a-4726-9b2e-a910d328e7f9-kube-api-access-4bp4x\") pod \"payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b\" (UID: \"a8aa349b-e04a-4726-9b2e-a910d328e7f9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b" Oct 29 23:37:48.041221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:48.035651 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bp4x\" (UniqueName: \"kubernetes.io/projected/a8aa349b-e04a-4726-9b2e-a910d328e7f9-kube-api-access-4bp4x\") pod \"payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b\" (UID: \"a8aa349b-e04a-4726-9b2e-a910d328e7f9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b" Oct 29 23:37:48.301572 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:48.301529 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b" Oct 29 23:37:48.524373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:48.524344 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw"] Oct 29 23:37:48.645267 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:37:48.645237 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8aa349b_e04a_4726_9b2e_a910d328e7f9.slice/crio-86e506f2c6ec24872b46d0473fce7005903b7b5e1ea93361414b51b72a7c35cf WatchSource:0}: Error finding container 86e506f2c6ec24872b46d0473fce7005903b7b5e1ea93361414b51b72a7c35cf: Status 404 returned error can't find the container with id 86e506f2c6ec24872b46d0473fce7005903b7b5e1ea93361414b51b72a7c35cf Oct 29 23:37:48.645476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:48.645457 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b"] Oct 29 23:37:49.035073 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:49.035031 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw" event={"ID":"ed4c81ea-a9b7-43c6-a893-342cd3d4389f","Type":"ContainerStarted","Data":"c7ed05d5970021f0ea51b7d5fd9456213ffe5c8f73d5e1eafe864750c8607ce2"} Oct 29 23:37:49.035598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:49.035102 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw" event={"ID":"ed4c81ea-a9b7-43c6-a893-342cd3d4389f","Type":"ContainerStarted","Data":"0c28f09290f195a104ec2ba1639fb46971682203510b4c33a142d844c37d2082"} Oct 29 23:37:49.035598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:49.035134 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw" Oct 29 23:37:49.037073 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:49.037046 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b" event={"ID":"a8aa349b-e04a-4726-9b2e-a910d328e7f9","Type":"ContainerStarted","Data":"baa540b9867d14046559e73916cb37c00e597b42420462eea5f7a2011f7a72a7"} Oct 29 23:37:49.037223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:49.037094 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b" event={"ID":"a8aa349b-e04a-4726-9b2e-a910d328e7f9","Type":"ContainerStarted","Data":"86e506f2c6ec24872b46d0473fce7005903b7b5e1ea93361414b51b72a7c35cf"} Oct 29 23:37:49.037223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:49.037124 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b" Oct 29 23:37:49.082619 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:49.082546 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw" podStartSLOduration=2.082529569 podStartE2EDuration="2.082529569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:37:49.075734086 +0000 UTC m=+3315.727232617" watchObservedRunningTime="2026-10-29 23:37:49.082529569 +0000 UTC m=+3315.734028101" Oct 29 23:37:49.121171 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:49.121125 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b" podStartSLOduration=2.12111203 podStartE2EDuration="2.12111203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:37:49.117473091 +0000 UTC m=+3315.768971620" watchObservedRunningTime="2026-10-29 23:37:49.12111203 +0000 UTC m=+3315.772610560" Oct 29 23:37:51.316411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:51.316378 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zm2vt" Oct 29 23:37:59.486087 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:59.486055 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j"] Oct 29 23:37:59.492481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:59.492461 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j" Oct 29 23:37:59.504539 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:59.502531 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j"] Oct 29 23:37:59.508006 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:59.507979 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-750972-dockercfg-wqf2k\"" Oct 29 23:37:59.611208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:59.610622 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-750972\"" Oct 29 23:37:59.849263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:59.839451 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf2zb\" (UniqueName: \"kubernetes.io/projected/27d88842-3d4c-436a-b613-67e561423024-kube-api-access-cf2zb\") pod \"payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j\" (UID: \"27d88842-3d4c-436a-b613-67e561423024\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j" Oct 29 23:37:59.849263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:59.839794 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27d88842-3d4c-436a-b613-67e561423024-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j\" (UID: \"27d88842-3d4c-436a-b613-67e561423024\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j" Oct 29 23:38:00.008211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:59.991883 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf2zb\" (UniqueName: \"kubernetes.io/projected/27d88842-3d4c-436a-b613-67e561423024-kube-api-access-cf2zb\") pod \"payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j\" (UID: \"27d88842-3d4c-436a-b613-67e561423024\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j" Oct 29 23:38:00.008211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:37:59.992394 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27d88842-3d4c-436a-b613-67e561423024-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j\" (UID: \"27d88842-3d4c-436a-b613-67e561423024\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j" Oct 29 23:38:00.031270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:00.015634 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27d88842-3d4c-436a-b613-67e561423024-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j\" (UID: \"27d88842-3d4c-436a-b613-67e561423024\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j" Oct 29 23:38:00.053962 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:00.052500 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw" Oct 29 23:38:00.069795 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:00.055606 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b" Oct 29 23:38:00.071240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:00.070597 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf2zb\" (UniqueName: \"kubernetes.io/projected/27d88842-3d4c-436a-b613-67e561423024-kube-api-access-cf2zb\") pod \"payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j\" (UID: \"27d88842-3d4c-436a-b613-67e561423024\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j" Oct 29 23:38:00.124985 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:00.124899 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j" Oct 29 23:38:00.448454 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:00.448427 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j"] Oct 29 23:38:00.449302 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:38:00.449264 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d88842_3d4c_436a_b613_67e561423024.slice/crio-cd3dbb91b91580588de878c8abf69c65bb19d89f0712bd230f90f372d07b2645 WatchSource:0}: Error finding container cd3dbb91b91580588de878c8abf69c65bb19d89f0712bd230f90f372d07b2645: Status 404 returned error can't find the container with id cd3dbb91b91580588de878c8abf69c65bb19d89f0712bd230f90f372d07b2645 Oct 29 23:38:01.165900 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:01.165865 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j" event={"ID":"27d88842-3d4c-436a-b613-67e561423024","Type":"ContainerStarted","Data":"e7e3c2454b50d3d5fa62a37e85dc8e04244d160a5bba3bc3ba04363b735585f2"} Oct 29 23:38:01.166364 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:01.165958 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j" Oct 29 23:38:01.166364 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:01.165977 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j" event={"ID":"27d88842-3d4c-436a-b613-67e561423024","Type":"ContainerStarted","Data":"cd3dbb91b91580588de878c8abf69c65bb19d89f0712bd230f90f372d07b2645"} Oct 29 23:38:01.211518 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:01.211462 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j" podStartSLOduration=2.211447557 podStartE2EDuration="2.211447557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:38:01.206339608 +0000 UTC m=+3327.857838138" watchObservedRunningTime="2026-10-29 23:38:01.211447557 +0000 UTC m=+3327.862946088" Oct 29 23:38:03.916296 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:38:03.916253 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:38:11.407474 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:11.407438 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4"] Oct 29 23:38:11.409151 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:11.409114 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" podUID="44918f77-9e08-4a3e-ba96-95763437e8ea" containerName="istio-proxy" containerID="cri-o://04be180318370ab58a2bcf850febb64bfc735bac26cbc8e27a599c3e1df678e7" gracePeriod=30 Oct 29 23:38:11.418082 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:11.418048 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml"] Oct 29 23:38:11.418429 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:11.418399 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" podUID="a846f9dc-451b-48f6-b370-cdd5276667ed" containerName="istio-proxy" containerID="cri-o://811248c3b1cbbc27cb71f15927e6f256d192b6ee4bfa502a7875b44f8b97165b" gracePeriod=30 Oct 29 23:38:11.424072 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:11.424036 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k"] Oct 29 23:38:11.424242 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:11.424220 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" podUID="3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" containerName="istio-proxy" containerID="cri-o://b2710aba657110990e9a1a8198fe8c23e45357fad2df5697e0026562d17a8a36" gracePeriod=30 Oct 29 23:38:12.189472 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:12.189441 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j" Oct 29 23:38:16.801505 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.801475 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:38:16.846293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.846253 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a846f9dc-451b-48f6-b370-cdd5276667ed-kube-api-access-qshx8\" (UniqueName: \"kubernetes.io/projected/a846f9dc-451b-48f6-b370-cdd5276667ed-kube-api-access-qshx8\") pod \"a846f9dc-451b-48f6-b370-cdd5276667ed\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " Oct 29 23:38:16.846491 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.846357 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-podinfo\") pod \"a846f9dc-451b-48f6-b370-cdd5276667ed\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " Oct 29 23:38:16.846491 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.846423 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a846f9dc-451b-48f6-b370-cdd5276667ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a846f9dc-451b-48f6-b370-cdd5276667ed-istiod-ca-cert\") pod \"a846f9dc-451b-48f6-b370-cdd5276667ed\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " Oct 29 23:38:16.846736 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.846711 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-credential-socket\") pod \"a846f9dc-451b-48f6-b370-cdd5276667ed\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " Oct 29 23:38:16.846822 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.846767 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-data\") pod \"a846f9dc-451b-48f6-b370-cdd5276667ed\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " Oct 29 23:38:16.846885 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.846854 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-token\" (UniqueName: \"kubernetes.io/projected/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-token\") pod \"a846f9dc-451b-48f6-b370-cdd5276667ed\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " Oct 29 23:38:16.846941 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.846917 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-envoy\") pod \"a846f9dc-451b-48f6-b370-cdd5276667ed\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " Oct 29 23:38:16.847052 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.847034 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-workload-socket\") pod \"a846f9dc-451b-48f6-b370-cdd5276667ed\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " Oct 29 23:38:16.847120 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.847086 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-workload-certs\") pod \"a846f9dc-451b-48f6-b370-cdd5276667ed\" (UID: \"a846f9dc-451b-48f6-b370-cdd5276667ed\") " Oct 29 23:38:16.849841 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.849473 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-data" pod "a846f9dc-451b-48f6-b370-cdd5276667ed" (UID: "a846f9dc-451b-48f6-b370-cdd5276667ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:16.853772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.853720 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-credential-socket" pod "a846f9dc-451b-48f6-b370-cdd5276667ed" (UID: "a846f9dc-451b-48f6-b370-cdd5276667ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:16.857479 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.857450 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a846f9dc-451b-48f6-b370-cdd5276667ed-istiod-ca-cert" pod "a846f9dc-451b-48f6-b370-cdd5276667ed" (UID: "a846f9dc-451b-48f6-b370-cdd5276667ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:38:16.857729 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.857697 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-workload-socket" pod "a846f9dc-451b-48f6-b370-cdd5276667ed" (UID: "a846f9dc-451b-48f6-b370-cdd5276667ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:16.877240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.869037 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-workload-certs" pod "a846f9dc-451b-48f6-b370-cdd5276667ed" (UID: "a846f9dc-451b-48f6-b370-cdd5276667ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:16.877240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.869597 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:38:16.877240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.872161 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a846f9dc-451b-48f6-b370-cdd5276667ed-kube-api-access-qshx8" pod "a846f9dc-451b-48f6-b370-cdd5276667ed" (UID: "a846f9dc-451b-48f6-b370-cdd5276667ed"). InnerVolumeSpecName "kube-api-access-qshx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:38:16.889247 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.884833 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-envoy" pod "a846f9dc-451b-48f6-b370-cdd5276667ed" (UID: "a846f9dc-451b-48f6-b370-cdd5276667ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:16.889247 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.885292 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-podinfo" pod "a846f9dc-451b-48f6-b370-cdd5276667ed" (UID: "a846f9dc-451b-48f6-b370-cdd5276667ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:38:16.889247 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.886382 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-token" pod "a846f9dc-451b-48f6-b370-cdd5276667ed" (UID: "a846f9dc-451b-48f6-b370-cdd5276667ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:38:16.912317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.912280 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:38:16.951740 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.951702 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-workload-certs\") pod \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " Oct 29 23:38:16.951913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.951764 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-podinfo\") pod \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " Oct 29 23:38:16.951913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.951820 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-kube-api-access-9wm8r\" (UniqueName: \"kubernetes.io/projected/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-kube-api-access-9wm8r\") pod \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " Oct 29 23:38:16.952058 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.951910 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-workload-socket\") pod \"44918f77-9e08-4a3e-ba96-95763437e8ea\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " Oct 29 23:38:16.952058 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.951979 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-workload-certs\") pod \"44918f77-9e08-4a3e-ba96-95763437e8ea\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " Oct 29 23:38:16.952058 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.952023 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-token\" (UniqueName: \"kubernetes.io/projected/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-token\") pod \"44918f77-9e08-4a3e-ba96-95763437e8ea\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " Oct 29 23:38:16.952273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.952125 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istiod-ca-cert\") pod \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " Oct 29 23:38:16.952273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.952155 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-podinfo\") pod \"44918f77-9e08-4a3e-ba96-95763437e8ea\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " Oct 29 23:38:16.952273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.952232 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-credential-socket\") pod \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " Oct 29 23:38:16.952443 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.952353 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-data\") pod \"44918f77-9e08-4a3e-ba96-95763437e8ea\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " Oct 29 23:38:16.952443 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.952420 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-token\" (UniqueName: \"kubernetes.io/projected/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-token\") pod \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " Oct 29 23:38:16.952555 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.952465 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-data\") pod \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " Oct 29 23:38:16.952617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.952561 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-envoy\") pod \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " Oct 29 23:38:16.952617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.952602 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-envoy\") pod \"44918f77-9e08-4a3e-ba96-95763437e8ea\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " Oct 29 23:38:16.952728 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.952638 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-credential-socket\") pod \"44918f77-9e08-4a3e-ba96-95763437e8ea\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " Oct 29 23:38:16.952728 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.952711 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44918f77-9e08-4a3e-ba96-95763437e8ea-kube-api-access-zcsg2\" (UniqueName: \"kubernetes.io/projected/44918f77-9e08-4a3e-ba96-95763437e8ea-kube-api-access-zcsg2\") pod \"44918f77-9e08-4a3e-ba96-95763437e8ea\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " Oct 29 23:38:16.952841 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.952794 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-workload-socket\") pod \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\" (UID: \"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f\") " Oct 29 23:38:16.952841 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.952825 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44918f77-9e08-4a3e-ba96-95763437e8ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44918f77-9e08-4a3e-ba96-95763437e8ea-istiod-ca-cert\") pod \"44918f77-9e08-4a3e-ba96-95763437e8ea\" (UID: \"44918f77-9e08-4a3e-ba96-95763437e8ea\") " Oct 29 23:38:16.953868 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.953838 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:16.953868 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.953868 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:16.954066 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.953882 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:16.954066 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.953895 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:16.954066 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.953910 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:16.954066 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.953923 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a846f9dc-451b-48f6-b370-cdd5276667ed-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:16.954066 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.953937 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qshx8\" (UniqueName: \"kubernetes.io/projected/a846f9dc-451b-48f6-b370-cdd5276667ed-kube-api-access-qshx8\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:16.954066 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.953951 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a846f9dc-451b-48f6-b370-cdd5276667ed-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:16.954066 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.953965 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a846f9dc-451b-48f6-b370-cdd5276667ed-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:16.954599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.954567 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-workload-socket" pod "44918f77-9e08-4a3e-ba96-95763437e8ea" (UID: "44918f77-9e08-4a3e-ba96-95763437e8ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:16.954826 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.954804 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-workload-certs" pod "3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" (UID: "3df5f43c-d322-4e8a-bb0f-ae13dc90e54f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:16.955804 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.955777 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-credential-socket" pod "3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" (UID: "3df5f43c-d322-4e8a-bb0f-ae13dc90e54f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:16.956239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.956209 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istiod-ca-cert" pod "3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" (UID: "3df5f43c-d322-4e8a-bb0f-ae13dc90e54f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:38:16.956436 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.956411 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44918f77-9e08-4a3e-ba96-95763437e8ea-istiod-ca-cert" pod "44918f77-9e08-4a3e-ba96-95763437e8ea" (UID: "44918f77-9e08-4a3e-ba96-95763437e8ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:38:16.957171 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.957144 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-data" pod "44918f77-9e08-4a3e-ba96-95763437e8ea" (UID: "44918f77-9e08-4a3e-ba96-95763437e8ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:16.958714 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.958688 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-workload-certs" pod "44918f77-9e08-4a3e-ba96-95763437e8ea" (UID: "44918f77-9e08-4a3e-ba96-95763437e8ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:16.958908 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.958886 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-workload-socket" pod "3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" (UID: "3df5f43c-d322-4e8a-bb0f-ae13dc90e54f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:16.959100 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.959080 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-credential-socket" pod "44918f77-9e08-4a3e-ba96-95763437e8ea" (UID: "44918f77-9e08-4a3e-ba96-95763437e8ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:16.959300 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.959279 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-data" pod "3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" (UID: "3df5f43c-d322-4e8a-bb0f-ae13dc90e54f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:16.965139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.965118 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44918f77-9e08-4a3e-ba96-95763437e8ea-kube-api-access-zcsg2" pod "44918f77-9e08-4a3e-ba96-95763437e8ea" (UID: "44918f77-9e08-4a3e-ba96-95763437e8ea"). InnerVolumeSpecName "kube-api-access-zcsg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:38:16.965634 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.965599 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-token" pod "44918f77-9e08-4a3e-ba96-95763437e8ea" (UID: "44918f77-9e08-4a3e-ba96-95763437e8ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:38:16.965739 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.965623 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-envoy" pod "44918f77-9e08-4a3e-ba96-95763437e8ea" (UID: "44918f77-9e08-4a3e-ba96-95763437e8ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:16.966318 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.965821 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-podinfo" pod "3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" (UID: "3df5f43c-d322-4e8a-bb0f-ae13dc90e54f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:38:16.966418 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.966335 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-envoy" pod "3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" (UID: "3df5f43c-d322-4e8a-bb0f-ae13dc90e54f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:16.966498 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.966463 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-kube-api-access-9wm8r" pod "3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" (UID: "3df5f43c-d322-4e8a-bb0f-ae13dc90e54f"). InnerVolumeSpecName "kube-api-access-9wm8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:38:16.966605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.966586 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-podinfo" pod "44918f77-9e08-4a3e-ba96-95763437e8ea" (UID: "44918f77-9e08-4a3e-ba96-95763437e8ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:38:16.966857 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:16.966836 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-token" pod "3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" (UID: "3df5f43c-d322-4e8a-bb0f-ae13dc90e54f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:38:17.056252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.056163 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:17.056252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.056204 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:17.056252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.056217 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:17.056252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.056229 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:17.056252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.056237 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:17.056252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.056245 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:17.056252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.056255 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcsg2\" (UniqueName: \"kubernetes.io/projected/44918f77-9e08-4a3e-ba96-95763437e8ea-kube-api-access-zcsg2\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:17.056604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.056264 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:17.056604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.056274 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44918f77-9e08-4a3e-ba96-95763437e8ea-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:17.056604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.056282 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:17.056604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.056291 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:17.056604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.056303 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wm8r\" (UniqueName: \"kubernetes.io/projected/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-kube-api-access-9wm8r\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:17.056604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.056311 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:17.056604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.056319 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44918f77-9e08-4a3e-ba96-95763437e8ea-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:17.056604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.056329 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:17.056604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.056336 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:17.056604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.056344 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44918f77-9e08-4a3e-ba96-95763437e8ea-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:17.056604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.056351 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:38:17.335926 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.335830 5057 generic.go:356] "Generic (PLEG): container finished" podID="44918f77-9e08-4a3e-ba96-95763437e8ea" containerID="04be180318370ab58a2bcf850febb64bfc735bac26cbc8e27a599c3e1df678e7" exitCode=0 Oct 29 23:38:17.335926 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.335886 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" Oct 29 23:38:17.336249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.335919 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" event={"ID":"44918f77-9e08-4a3e-ba96-95763437e8ea","Type":"ContainerDied","Data":"04be180318370ab58a2bcf850febb64bfc735bac26cbc8e27a599c3e1df678e7"} Oct 29 23:38:17.336249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.336012 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4" event={"ID":"44918f77-9e08-4a3e-ba96-95763437e8ea","Type":"ContainerDied","Data":"7c7dbb04d1f86a090188c5025cef4dbfce142aae35a2830b817fe9c70a2772c4"} Oct 29 23:38:17.336249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.336041 5057 scope.go:122] "RemoveContainer" containerID="04be180318370ab58a2bcf850febb64bfc735bac26cbc8e27a599c3e1df678e7" Oct 29 23:38:17.337765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.337740 5057 generic.go:356] "Generic (PLEG): container finished" podID="a846f9dc-451b-48f6-b370-cdd5276667ed" containerID="811248c3b1cbbc27cb71f15927e6f256d192b6ee4bfa502a7875b44f8b97165b" exitCode=0 Oct 29 23:38:17.337967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.337783 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" event={"ID":"a846f9dc-451b-48f6-b370-cdd5276667ed","Type":"ContainerDied","Data":"811248c3b1cbbc27cb71f15927e6f256d192b6ee4bfa502a7875b44f8b97165b"} Oct 29 23:38:17.338086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.337999 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" event={"ID":"a846f9dc-451b-48f6-b370-cdd5276667ed","Type":"ContainerDied","Data":"25e4dc835f183e8285b6ebba02c04100d441df04bbd495ea0a6d123ee432a4f5"} Oct 29 23:38:17.338086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.337807 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml" Oct 29 23:38:17.345276 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.345252 5057 generic.go:356] "Generic (PLEG): container finished" podID="3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" containerID="b2710aba657110990e9a1a8198fe8c23e45357fad2df5697e0026562d17a8a36" exitCode=0 Oct 29 23:38:17.345490 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.345289 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" event={"ID":"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f","Type":"ContainerDied","Data":"b2710aba657110990e9a1a8198fe8c23e45357fad2df5697e0026562d17a8a36"} Oct 29 23:38:17.345578 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.345525 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" event={"ID":"3df5f43c-d322-4e8a-bb0f-ae13dc90e54f","Type":"ContainerDied","Data":"3da8692efc4df9fee4b445de0b7a3d234feaac97db6989fdc3d020445e8266fa"} Oct 29 23:38:17.345578 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.345373 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k" Oct 29 23:38:17.355135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.354983 5057 scope.go:122] "RemoveContainer" containerID="04be180318370ab58a2bcf850febb64bfc735bac26cbc8e27a599c3e1df678e7" Oct 29 23:38:17.355392 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:38:17.355368 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04be180318370ab58a2bcf850febb64bfc735bac26cbc8e27a599c3e1df678e7\": container with ID starting with 04be180318370ab58a2bcf850febb64bfc735bac26cbc8e27a599c3e1df678e7 not found: ID does not exist" containerID="04be180318370ab58a2bcf850febb64bfc735bac26cbc8e27a599c3e1df678e7" Oct 29 23:38:17.355610 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.355579 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04be180318370ab58a2bcf850febb64bfc735bac26cbc8e27a599c3e1df678e7"} err="failed to get container status \"04be180318370ab58a2bcf850febb64bfc735bac26cbc8e27a599c3e1df678e7\": rpc error: code = NotFound desc = could not find container \"04be180318370ab58a2bcf850febb64bfc735bac26cbc8e27a599c3e1df678e7\": container with ID starting with 04be180318370ab58a2bcf850febb64bfc735bac26cbc8e27a599c3e1df678e7 not found: ID does not exist" Oct 29 23:38:17.355713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.355614 5057 scope.go:122] "RemoveContainer" containerID="811248c3b1cbbc27cb71f15927e6f256d192b6ee4bfa502a7875b44f8b97165b" Oct 29 23:38:17.369820 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.369801 5057 scope.go:122] "RemoveContainer" containerID="811248c3b1cbbc27cb71f15927e6f256d192b6ee4bfa502a7875b44f8b97165b" Oct 29 23:38:17.370135 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:38:17.370115 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"811248c3b1cbbc27cb71f15927e6f256d192b6ee4bfa502a7875b44f8b97165b\": container with ID starting with 811248c3b1cbbc27cb71f15927e6f256d192b6ee4bfa502a7875b44f8b97165b not found: ID does not exist" containerID="811248c3b1cbbc27cb71f15927e6f256d192b6ee4bfa502a7875b44f8b97165b" Oct 29 23:38:17.370220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.370142 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"811248c3b1cbbc27cb71f15927e6f256d192b6ee4bfa502a7875b44f8b97165b"} err="failed to get container status \"811248c3b1cbbc27cb71f15927e6f256d192b6ee4bfa502a7875b44f8b97165b\": rpc error: code = NotFound desc = could not find container \"811248c3b1cbbc27cb71f15927e6f256d192b6ee4bfa502a7875b44f8b97165b\": container with ID starting with 811248c3b1cbbc27cb71f15927e6f256d192b6ee4bfa502a7875b44f8b97165b not found: ID does not exist" Oct 29 23:38:17.370220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.370158 5057 scope.go:122] "RemoveContainer" containerID="b2710aba657110990e9a1a8198fe8c23e45357fad2df5697e0026562d17a8a36" Oct 29 23:38:17.381455 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.381422 5057 scope.go:122] "RemoveContainer" containerID="b2710aba657110990e9a1a8198fe8c23e45357fad2df5697e0026562d17a8a36" Oct 29 23:38:17.381728 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:38:17.381707 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2710aba657110990e9a1a8198fe8c23e45357fad2df5697e0026562d17a8a36\": container with ID starting with b2710aba657110990e9a1a8198fe8c23e45357fad2df5697e0026562d17a8a36 not found: ID does not exist" containerID="b2710aba657110990e9a1a8198fe8c23e45357fad2df5697e0026562d17a8a36" Oct 29 23:38:17.381791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.381735 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2710aba657110990e9a1a8198fe8c23e45357fad2df5697e0026562d17a8a36"} err="failed to get container status \"b2710aba657110990e9a1a8198fe8c23e45357fad2df5697e0026562d17a8a36\": rpc error: code = NotFound desc = could not find container \"b2710aba657110990e9a1a8198fe8c23e45357fad2df5697e0026562d17a8a36\": container with ID starting with b2710aba657110990e9a1a8198fe8c23e45357fad2df5697e0026562d17a8a36 not found: ID does not exist" Oct 29 23:38:17.427137 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.427089 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k"] Oct 29 23:38:17.437321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.437289 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-7vb5k"] Oct 29 23:38:17.485350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.485317 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" path="/var/lib/kubelet/pods/3df5f43c-d322-4e8a-bb0f-ae13dc90e54f/volumes" Oct 29 23:38:17.485882 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.485863 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml"] Oct 29 23:38:17.485956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.485904 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-c5jml"] Oct 29 23:38:17.510735 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.510701 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4"] Oct 29 23:38:17.515568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:17.515537 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-58ks4"] Oct 29 23:38:19.486491 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:19.486454 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44918f77-9e08-4a3e-ba96-95763437e8ea" path="/var/lib/kubelet/pods/44918f77-9e08-4a3e-ba96-95763437e8ea/volumes" Oct 29 23:38:19.487139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:19.487120 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a846f9dc-451b-48f6-b370-cdd5276667ed" path="/var/lib/kubelet/pods/a846f9dc-451b-48f6-b370-cdd5276667ed/volumes" Oct 29 23:38:25.183367 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:25.183335 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:38:28.603943 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:28.603893 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:38:28.786210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:28.782274 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:38:28.978330 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:28.978281 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" podUID="9033febc-c7ee-4884-9145-c27229f9f5dd" containerName="main" probeResult="failure" output="Get \"https://10.132.1.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:38:29.034210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:29.033305 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:38:29.034520 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:29.034485 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:38:33.917428 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:38:33.917382 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:38:47.128815 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.128733 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr"] Oct 29 23:38:47.129686 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.129670 5057 cpu_manager.go:414] "containerMap: removing container" podUID="44918f77-9e08-4a3e-ba96-95763437e8ea" containerName="istio-proxy" Oct 29 23:38:47.129766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.129687 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44918f77-9e08-4a3e-ba96-95763437e8ea" containerName="istio-proxy" Oct 29 23:38:47.129766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.129701 5057 cpu_manager.go:414] "containerMap: removing container" podUID="3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" containerName="istio-proxy" Oct 29 23:38:47.129766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.129707 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" containerName="istio-proxy" Oct 29 23:38:47.129766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.129725 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a846f9dc-451b-48f6-b370-cdd5276667ed" containerName="istio-proxy" Oct 29 23:38:47.129766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.129730 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a846f9dc-451b-48f6-b370-cdd5276667ed" containerName="istio-proxy" Oct 29 23:38:47.129966 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.129897 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="44918f77-9e08-4a3e-ba96-95763437e8ea" containerName="istio-proxy" Oct 29 23:38:47.129966 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.129908 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="3df5f43c-d322-4e8a-bb0f-ae13dc90e54f" containerName="istio-proxy" Oct 29 23:38:47.129966 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.129918 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a846f9dc-451b-48f6-b370-cdd5276667ed" containerName="istio-proxy" Oct 29 23:38:47.135478 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.135452 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" Oct 29 23:38:47.161736 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.161708 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr"] Oct 29 23:38:47.163410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.163383 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a8ab3c-serving-cert\"" Oct 29 23:38:47.211993 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.211945 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/634af89c-ecba-484c-8843-5154b3676abb-maas-api-tls\") pod \"maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr\" (UID: \"634af89c-ecba-484c-8843-5154b3676abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" Oct 29 23:38:47.212223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.212022 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwgw2\" (UniqueName: \"kubernetes.io/projected/634af89c-ecba-484c-8843-5154b3676abb-kube-api-access-bwgw2\") pod \"maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr\" (UID: \"634af89c-ecba-484c-8843-5154b3676abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" Oct 29 23:38:47.212313 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.212232 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/634af89c-ecba-484c-8843-5154b3676abb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr\" (UID: \"634af89c-ecba-484c-8843-5154b3676abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" Oct 29 23:38:47.212428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.212406 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/634af89c-ecba-484c-8843-5154b3676abb-metrics-certs\") pod \"maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr\" (UID: \"634af89c-ecba-484c-8843-5154b3676abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" Oct 29 23:38:47.313869 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.313835 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/634af89c-ecba-484c-8843-5154b3676abb-maas-api-tls\") pod \"maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr\" (UID: \"634af89c-ecba-484c-8843-5154b3676abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" Oct 29 23:38:47.314075 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.313891 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwgw2\" (UniqueName: \"kubernetes.io/projected/634af89c-ecba-484c-8843-5154b3676abb-kube-api-access-bwgw2\") pod \"maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr\" (UID: \"634af89c-ecba-484c-8843-5154b3676abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" Oct 29 23:38:47.314075 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.314001 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/634af89c-ecba-484c-8843-5154b3676abb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr\" (UID: \"634af89c-ecba-484c-8843-5154b3676abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" Oct 29 23:38:47.314270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.314149 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/634af89c-ecba-484c-8843-5154b3676abb-metrics-certs\") pod \"maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr\" (UID: \"634af89c-ecba-484c-8843-5154b3676abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" Oct 29 23:38:47.314776 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.314748 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/634af89c-ecba-484c-8843-5154b3676abb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr\" (UID: \"634af89c-ecba-484c-8843-5154b3676abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" Oct 29 23:38:47.316609 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.316585 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/634af89c-ecba-484c-8843-5154b3676abb-maas-api-tls\") pod \"maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr\" (UID: \"634af89c-ecba-484c-8843-5154b3676abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" Oct 29 23:38:47.317042 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.317020 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/634af89c-ecba-484c-8843-5154b3676abb-metrics-certs\") pod \"maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr\" (UID: \"634af89c-ecba-484c-8843-5154b3676abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" Oct 29 23:38:47.338049 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.338019 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwgw2\" (UniqueName: \"kubernetes.io/projected/634af89c-ecba-484c-8843-5154b3676abb-kube-api-access-bwgw2\") pod \"maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr\" (UID: \"634af89c-ecba-484c-8843-5154b3676abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" Oct 29 23:38:47.471003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.470621 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" Oct 29 23:38:47.826909 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:38:47.826877 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634af89c_ecba_484c_8843_5154b3676abb.slice/crio-5e062f1a702f754078fc14c291456bf8aee9a48aa7952f360f9cbe127986ba1e WatchSource:0}: Error finding container 5e062f1a702f754078fc14c291456bf8aee9a48aa7952f360f9cbe127986ba1e: Status 404 returned error can't find the container with id 5e062f1a702f754078fc14c291456bf8aee9a48aa7952f360f9cbe127986ba1e Oct 29 23:38:47.829281 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.829250 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr"] Oct 29 23:38:47.934955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:47.934904 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" event={"ID":"634af89c-ecba-484c-8843-5154b3676abb","Type":"ContainerStarted","Data":"5e062f1a702f754078fc14c291456bf8aee9a48aa7952f360f9cbe127986ba1e"} Oct 29 23:38:48.948130 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:48.948082 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" event={"ID":"634af89c-ecba-484c-8843-5154b3676abb","Type":"ContainerStarted","Data":"c1edcea327342bfb5395ed68df45d0c37cdeca7657c323c527e3f631d3d2266d"} Oct 29 23:38:48.948627 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:48.948167 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" Oct 29 23:38:48.993957 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:48.993902 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" podStartSLOduration=1.6403434049999999 podStartE2EDuration="1.993885512s" totalImagesPullingTime="353.542107ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:38:48.989733706 +0000 UTC m=+3375.641232238" watchObservedRunningTime="2026-10-29 23:38:48.993885512 +0000 UTC m=+3375.645384043" Oct 29 23:38:54.998112 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:54.998059 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" Oct 29 23:38:59.875461 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:59.875422 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg"] Oct 29 23:38:59.881999 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:59.881971 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" Oct 29 23:38:59.910071 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:59.910039 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3b8149-serving-cert\"" Oct 29 23:38:59.917631 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:59.917595 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg"] Oct 29 23:38:59.944850 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:59.944814 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8s86\" (UniqueName: \"kubernetes.io/projected/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-kube-api-access-q8s86\") pod \"maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg\" (UID: \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" Oct 29 23:38:59.945070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:59.944932 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-metrics-certs\") pod \"maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg\" (UID: \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" Oct 29 23:38:59.945147 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:59.945080 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-maas-api-tls\") pod \"maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg\" (UID: \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" Oct 29 23:38:59.945263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:38:59.945214 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg\" (UID: \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" Oct 29 23:39:00.047468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:00.047412 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-maas-api-tls\") pod \"maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg\" (UID: \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" Oct 29 23:39:00.048467 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:39:00.047536 5057 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-serving-cert: secret "maas-api-e2e-inf-a-3b8149-serving-cert" not found Oct 29 23:39:00.048467 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:39:00.047591 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-maas-api-tls podName:23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1 nodeName:}" failed. No retries permitted until 2026-10-29 23:39:00.547577153 +0000 UTC m=+3387.199075664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-maas-api-tls") pod "maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" (UID: "23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1") : secret "maas-api-e2e-inf-a-3b8149-serving-cert" not found Oct 29 23:39:00.048467 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:00.047541 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg\" (UID: \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" Oct 29 23:39:00.048467 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:00.047933 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8s86\" (UniqueName: \"kubernetes.io/projected/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-kube-api-access-q8s86\") pod \"maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg\" (UID: \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" Oct 29 23:39:00.048467 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:00.048027 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-metrics-certs\") pod \"maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg\" (UID: \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" Oct 29 23:39:00.048467 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:00.048288 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg\" (UID: \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" Oct 29 23:39:00.114749 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:00.102112 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-metrics-certs\") pod \"maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg\" (UID: \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" Oct 29 23:39:00.114749 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:00.114599 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8s86\" (UniqueName: \"kubernetes.io/projected/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-kube-api-access-q8s86\") pod \"maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg\" (UID: \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" Oct 29 23:39:00.583320 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:00.583283 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-maas-api-tls\") pod \"maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg\" (UID: \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" Oct 29 23:39:00.586599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:00.586555 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-maas-api-tls\") pod \"maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg\" (UID: \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" Oct 29 23:39:00.813972 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:00.813749 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" Oct 29 23:39:01.316975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:01.316938 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg"] Oct 29 23:39:02.330598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.330546 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" event={"ID":"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1","Type":"ContainerStarted","Data":"f91c81529f6b9fe99e4812e82d93b35b41c80e7e48fd9590778a61b68172001c"} Oct 29 23:39:02.331101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.330611 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" event={"ID":"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1","Type":"ContainerStarted","Data":"6709d4262a0236a6b62ff88cb3d5354683d45b7c88c4db19838b374a1379cfb6"} Oct 29 23:39:02.331101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.330641 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" Oct 29 23:39:02.413920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.413881 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr"] Oct 29 23:39:02.417554 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.417501 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" podStartSLOduration=3.065340526 podStartE2EDuration="3.417487409s" totalImagesPullingTime="352.146883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:38:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:39:02.405309935 +0000 UTC m=+3389.056808466" watchObservedRunningTime="2026-10-29 23:39:02.417487409 +0000 UTC m=+3389.068985939" Oct 29 23:39:02.421022 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.420995 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.441064 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.440997 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-37498a-openshift-default-dockercfg-pjlnk\"" Oct 29 23:39:02.452176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.451710 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr"] Oct 29 23:39:02.568212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.552613 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-podinfo\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.593512 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.593372 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-workload-certs\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.593842 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.593589 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-credential-socket\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.596215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.594621 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-data\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.616324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.616284 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-envoy\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.618092 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.618060 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsp2b\" (UniqueName: \"kubernetes.io/projected/a28590ff-f920-4993-b5f2-d8afb67b3685-kube-api-access-tsp2b\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.618454 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.618432 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-token\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.618644 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.618621 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-workload-socket\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.618970 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.618947 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28590ff-f920-4993-b5f2-d8afb67b3685-istiod-ca-cert\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.755328 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.748661 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-envoy\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.755328 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.748754 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsp2b\" (UniqueName: \"kubernetes.io/projected/a28590ff-f920-4993-b5f2-d8afb67b3685-kube-api-access-tsp2b\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.755328 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.748861 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-token\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.755328 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.748906 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-workload-socket\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.755328 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.749085 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28590ff-f920-4993-b5f2-d8afb67b3685-istiod-ca-cert\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.755328 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.749149 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-podinfo\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.755328 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.749217 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-workload-certs\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.755328 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.749263 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-credential-socket\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.755328 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.749361 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-data\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.785211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.784495 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28590ff-f920-4993-b5f2-d8afb67b3685-istiod-ca-cert\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.809599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.792539 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-envoy\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.809599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.792841 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-workload-certs\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.809599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.805579 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-workload-socket\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.809599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.805842 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-credential-socket\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.809599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.807509 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-data\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.827349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.827270 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsp2b\" (UniqueName: \"kubernetes.io/projected/a28590ff-f920-4993-b5f2-d8afb67b3685-kube-api-access-tsp2b\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.841487 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.840896 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-podinfo\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:02.880924 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:02.880794 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-token\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:03.049236 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:03.048464 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:03.387970 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:39:03.387908 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda28590ff_f920_4993_b5f2_d8afb67b3685.slice/crio-ddf9b9a8cd7644cd4091d4de49019500c5018571ca1ff7126ccb0eec8861ce92 WatchSource:0}: Error finding container ddf9b9a8cd7644cd4091d4de49019500c5018571ca1ff7126ccb0eec8861ce92: Status 404 returned error can't find the container with id ddf9b9a8cd7644cd4091d4de49019500c5018571ca1ff7126ccb0eec8861ce92 Oct 29 23:39:03.388459 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:03.388435 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr"] Oct 29 23:39:03.391653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:03.391620 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:39:03.391733 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:03.391685 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:39:03.391733 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:03.391717 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:39:03.963950 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:39:03.963909 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:39:04.350133 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:04.350045 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" event={"ID":"a28590ff-f920-4993-b5f2-d8afb67b3685","Type":"ContainerStarted","Data":"3d2dadd12f15bedf8f8b5865e19c4f94f020298f717606fa2fb16a7a4289ce8c"} Oct 29 23:39:04.350133 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:04.350128 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" event={"ID":"a28590ff-f920-4993-b5f2-d8afb67b3685","Type":"ContainerStarted","Data":"ddf9b9a8cd7644cd4091d4de49019500c5018571ca1ff7126ccb0eec8861ce92"} Oct 29 23:39:04.413633 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:04.413574 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" podStartSLOduration=2.413557322 podStartE2EDuration="2.413557322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:39:04.407948798 +0000 UTC m=+3391.059447341" watchObservedRunningTime="2026-10-29 23:39:04.413557322 +0000 UTC m=+3391.065055851" Oct 29 23:39:05.049546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:05.049510 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:05.056350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:05.056322 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:05.441212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:05.439726 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:05.457585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:05.457555 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:39:08.347114 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:08.347087 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" Oct 29 23:39:08.892468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:08.892429 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8"] Oct 29 23:39:08.898294 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:08.898262 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8" Oct 29 23:39:08.916114 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:08.916085 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-37498a\"" Oct 29 23:39:08.921482 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:08.921453 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8"] Oct 29 23:39:08.927291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:08.927269 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-37498a-dockercfg-ddngq\"" Oct 29 23:39:09.085212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:09.055098 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4494b17b-7e7b-41e4-a759-0284b87f5a57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8\" (UID: \"4494b17b-7e7b-41e4-a759-0284b87f5a57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8" Oct 29 23:39:09.098501 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:09.098458 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjqs5\" (UniqueName: \"kubernetes.io/projected/4494b17b-7e7b-41e4-a759-0284b87f5a57-kube-api-access-bjqs5\") pod \"payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8\" (UID: \"4494b17b-7e7b-41e4-a759-0284b87f5a57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8" Oct 29 23:39:09.246634 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:09.244494 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjqs5\" (UniqueName: \"kubernetes.io/projected/4494b17b-7e7b-41e4-a759-0284b87f5a57-kube-api-access-bjqs5\") pod \"payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8\" (UID: \"4494b17b-7e7b-41e4-a759-0284b87f5a57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8" Oct 29 23:39:09.248157 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:09.247290 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4494b17b-7e7b-41e4-a759-0284b87f5a57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8\" (UID: \"4494b17b-7e7b-41e4-a759-0284b87f5a57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8" Oct 29 23:39:09.248765 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:09.248725 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4494b17b-7e7b-41e4-a759-0284b87f5a57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8\" (UID: \"4494b17b-7e7b-41e4-a759-0284b87f5a57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8" Oct 29 23:39:09.394571 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:09.394535 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjqs5\" (UniqueName: \"kubernetes.io/projected/4494b17b-7e7b-41e4-a759-0284b87f5a57-kube-api-access-bjqs5\") pod \"payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8\" (UID: \"4494b17b-7e7b-41e4-a759-0284b87f5a57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8" Oct 29 23:39:09.515249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:09.515157 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8" Oct 29 23:39:09.845885 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:39:09.845841 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4494b17b_7e7b_41e4_a759_0284b87f5a57.slice/crio-2a0dfd54adffb546021a584c7f767eec6703f695b065a1f5cd6c699e14069e8d WatchSource:0}: Error finding container 2a0dfd54adffb546021a584c7f767eec6703f695b065a1f5cd6c699e14069e8d: Status 404 returned error can't find the container with id 2a0dfd54adffb546021a584c7f767eec6703f695b065a1f5cd6c699e14069e8d Oct 29 23:39:09.846967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:09.846937 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8"] Oct 29 23:39:10.524712 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:10.524671 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8" event={"ID":"4494b17b-7e7b-41e4-a759-0284b87f5a57","Type":"ContainerStarted","Data":"4e7dea9121622967d793a2dac0763d44a1e2811946cfe18e403cbe3eee9ab96e"} Oct 29 23:39:10.525257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:10.524732 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8" event={"ID":"4494b17b-7e7b-41e4-a759-0284b87f5a57","Type":"ContainerStarted","Data":"2a0dfd54adffb546021a584c7f767eec6703f695b065a1f5cd6c699e14069e8d"} Oct 29 23:39:10.525257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:10.524771 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8" Oct 29 23:39:10.567862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:10.567810 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8" podStartSLOduration=2.567797111 podStartE2EDuration="2.567797111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:39:10.565123356 +0000 UTC m=+3397.216621891" watchObservedRunningTime="2026-10-29 23:39:10.567797111 +0000 UTC m=+3397.219295641" Oct 29 23:39:21.537415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:21.537382 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8" Oct 29 23:39:33.914204 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:39:33.914155 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:39:46.341218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:46.341157 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:39:49.487210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:49.483998 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:39:49.636228 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:49.634258 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" podUID="9033febc-c7ee-4884-9145-c27229f9f5dd" containerName="main" probeResult="failure" output="Get \"https://10.132.1.40:8000/health\": context deadline exceeded" Oct 29 23:39:49.858913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:49.854000 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 23:39:51.184910 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:51.184851 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf"] Oct 29 23:39:56.185261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:56.185221 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf" podUID="483d8088-c440-44da-ab9e-b7f4edd88410" containerName="payload-pre-processing" containerID="cri-o://7af80cd6829641f615214cb1924bfe8e2dcd7846098a0bea85a7c521a682a069" gracePeriod=25 Oct 29 23:39:56.360857 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:56.360819 5057 generic.go:356] "Generic (PLEG): container finished" podID="483d8088-c440-44da-ab9e-b7f4edd88410" containerID="7af80cd6829641f615214cb1924bfe8e2dcd7846098a0bea85a7c521a682a069" exitCode=0 Oct 29 23:39:56.361050 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:56.360893 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf" event={"ID":"483d8088-c440-44da-ab9e-b7f4edd88410","Type":"ContainerDied","Data":"7af80cd6829641f615214cb1924bfe8e2dcd7846098a0bea85a7c521a682a069"} Oct 29 23:39:56.517535 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:56.517509 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf" Oct 29 23:39:56.699378 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:56.699344 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/483d8088-c440-44da-ab9e-b7f4edd88410-kube-api-access-ljzkd\" (UniqueName: \"kubernetes.io/projected/483d8088-c440-44da-ab9e-b7f4edd88410-kube-api-access-ljzkd\") pod \"483d8088-c440-44da-ab9e-b7f4edd88410\" (UID: \"483d8088-c440-44da-ab9e-b7f4edd88410\") " Oct 29 23:39:56.699567 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:56.699449 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/483d8088-c440-44da-ab9e-b7f4edd88410-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/483d8088-c440-44da-ab9e-b7f4edd88410-plugins-config-volume\") pod \"483d8088-c440-44da-ab9e-b7f4edd88410\" (UID: \"483d8088-c440-44da-ab9e-b7f4edd88410\") " Oct 29 23:39:56.699845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:56.699821 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/483d8088-c440-44da-ab9e-b7f4edd88410-plugins-config-volume" pod "483d8088-c440-44da-ab9e-b7f4edd88410" (UID: "483d8088-c440-44da-ab9e-b7f4edd88410"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:39:56.700271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:56.700255 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/483d8088-c440-44da-ab9e-b7f4edd88410-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:39:56.701783 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:56.701764 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/483d8088-c440-44da-ab9e-b7f4edd88410-kube-api-access-ljzkd" pod "483d8088-c440-44da-ab9e-b7f4edd88410" (UID: "483d8088-c440-44da-ab9e-b7f4edd88410"). InnerVolumeSpecName "kube-api-access-ljzkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:39:56.801811 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:56.801731 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljzkd\" (UniqueName: \"kubernetes.io/projected/483d8088-c440-44da-ab9e-b7f4edd88410-kube-api-access-ljzkd\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:39:57.372173 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:57.372138 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf" Oct 29 23:39:57.372173 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:57.372159 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf" event={"ID":"483d8088-c440-44da-ab9e-b7f4edd88410","Type":"ContainerDied","Data":"908278476d6f99d9d1ce5592d51ac2914aadbbc97986b2c41d6e32c35beaa137"} Oct 29 23:39:57.372781 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:57.372257 5057 scope.go:122] "RemoveContainer" containerID="7af80cd6829641f615214cb1924bfe8e2dcd7846098a0bea85a7c521a682a069" Oct 29 23:39:57.429246 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:57.429182 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf"] Oct 29 23:39:57.443646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:57.443601 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-90f683-7cb7d7bff-whglf"] Oct 29 23:39:57.485171 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:39:57.485138 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="483d8088-c440-44da-ab9e-b7f4edd88410" path="/var/lib/kubelet/pods/483d8088-c440-44da-ab9e-b7f4edd88410/volumes" Oct 29 23:40:03.915438 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:40:03.915398 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:40:17.876962 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:17.876875 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4"] Oct 29 23:40:17.878141 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:17.878118 5057 cpu_manager.go:414] "containerMap: removing container" podUID="483d8088-c440-44da-ab9e-b7f4edd88410" containerName="payload-pre-processing" Oct 29 23:40:17.878141 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:17.878141 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="483d8088-c440-44da-ab9e-b7f4edd88410" containerName="payload-pre-processing" Oct 29 23:40:17.878346 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:17.878322 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="483d8088-c440-44da-ab9e-b7f4edd88410" containerName="payload-pre-processing" Oct 29 23:40:17.882571 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:17.882549 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4" Oct 29 23:40:17.902498 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:17.902472 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4"] Oct 29 23:40:17.909056 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:17.909035 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-55c3fc\"" Oct 29 23:40:17.909167 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:17.909089 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-55c3fc-dockercfg-jq5bv\"" Oct 29 23:40:18.056209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:18.050779 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pltm6\" (UniqueName: \"kubernetes.io/projected/b7efd7af-6379-4f18-b4e0-6ed403891e10-kube-api-access-pltm6\") pod \"payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4\" (UID: \"b7efd7af-6379-4f18-b4e0-6ed403891e10\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4" Oct 29 23:40:18.056209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:18.050844 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7efd7af-6379-4f18-b4e0-6ed403891e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4\" (UID: \"b7efd7af-6379-4f18-b4e0-6ed403891e10\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4" Oct 29 23:40:18.220262 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:18.173110 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pltm6\" (UniqueName: \"kubernetes.io/projected/b7efd7af-6379-4f18-b4e0-6ed403891e10-kube-api-access-pltm6\") pod \"payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4\" (UID: \"b7efd7af-6379-4f18-b4e0-6ed403891e10\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4" Oct 29 23:40:18.249332 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:18.249290 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7efd7af-6379-4f18-b4e0-6ed403891e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4\" (UID: \"b7efd7af-6379-4f18-b4e0-6ed403891e10\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4" Oct 29 23:40:18.280393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:18.251438 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7efd7af-6379-4f18-b4e0-6ed403891e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4\" (UID: \"b7efd7af-6379-4f18-b4e0-6ed403891e10\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4" Oct 29 23:40:18.418435 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:18.418383 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pltm6\" (UniqueName: \"kubernetes.io/projected/b7efd7af-6379-4f18-b4e0-6ed403891e10-kube-api-access-pltm6\") pod \"payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4\" (UID: \"b7efd7af-6379-4f18-b4e0-6ed403891e10\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4" Oct 29 23:40:18.497581 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:18.497499 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4" Oct 29 23:40:18.821864 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:40:18.821831 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7efd7af_6379_4f18_b4e0_6ed403891e10.slice/crio-1719f878a79f6fe10dd2e07db72549bcdefe10c1b417b6c2a15ef89b5e2c230e WatchSource:0}: Error finding container 1719f878a79f6fe10dd2e07db72549bcdefe10c1b417b6c2a15ef89b5e2c230e: Status 404 returned error can't find the container with id 1719f878a79f6fe10dd2e07db72549bcdefe10c1b417b6c2a15ef89b5e2c230e Oct 29 23:40:18.823107 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:18.823075 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4"] Oct 29 23:40:19.714560 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:19.714524 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4" event={"ID":"b7efd7af-6379-4f18-b4e0-6ed403891e10","Type":"ContainerStarted","Data":"cc4ce5889b8787157cbbf4ea5afdabd8192cd2e9240ab8e979d35561d7b21442"} Oct 29 23:40:19.714970 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:19.714577 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4" event={"ID":"b7efd7af-6379-4f18-b4e0-6ed403891e10","Type":"ContainerStarted","Data":"1719f878a79f6fe10dd2e07db72549bcdefe10c1b417b6c2a15ef89b5e2c230e"} Oct 29 23:40:19.714970 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:19.714769 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4" Oct 29 23:40:19.756944 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:19.756895 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4" podStartSLOduration=2.7568811699999998 podStartE2EDuration="2.75688117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:40:19.75472009 +0000 UTC m=+3466.406218620" watchObservedRunningTime="2026-10-29 23:40:19.75688117 +0000 UTC m=+3466.408379701" Oct 29 23:40:30.727249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:30.727204 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4" Oct 29 23:40:33.915872 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:40:33.915830 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:40:48.039177 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:48.039132 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:40:58.444313 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:40:58.444274 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w"] Oct 29 23:41:00.729213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:00.712053 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:41:03.446484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:03.446255 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w" podUID="2f2943d7-12ce-4123-a75b-bbbd49a823a5" containerName="payload-pre-processing" containerID="cri-o://fade27f70677cd84512a6f4825d53d321b69cde4464454db94007295b98b3fb4" gracePeriod=25 Oct 29 23:41:03.941148 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:41:03.941079 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:41:04.070270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:04.070215 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w" Oct 29 23:41:04.094706 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:04.094675 5057 generic.go:356] "Generic (PLEG): container finished" podID="2f2943d7-12ce-4123-a75b-bbbd49a823a5" containerID="fade27f70677cd84512a6f4825d53d321b69cde4464454db94007295b98b3fb4" exitCode=0 Oct 29 23:41:04.094887 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:04.094721 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w" Oct 29 23:41:04.094887 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:04.094767 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w" event={"ID":"2f2943d7-12ce-4123-a75b-bbbd49a823a5","Type":"ContainerDied","Data":"fade27f70677cd84512a6f4825d53d321b69cde4464454db94007295b98b3fb4"} Oct 29 23:41:04.094887 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:04.094816 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w" event={"ID":"2f2943d7-12ce-4123-a75b-bbbd49a823a5","Type":"ContainerDied","Data":"ced5298298089416bc0beae8f89336b5625f28e54d7f596f5fc738a00c153ee0"} Oct 29 23:41:04.094887 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:04.094840 5057 scope.go:122] "RemoveContainer" containerID="fade27f70677cd84512a6f4825d53d321b69cde4464454db94007295b98b3fb4" Oct 29 23:41:04.106641 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:04.106616 5057 scope.go:122] "RemoveContainer" containerID="fade27f70677cd84512a6f4825d53d321b69cde4464454db94007295b98b3fb4" Oct 29 23:41:04.106951 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:41:04.106925 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fade27f70677cd84512a6f4825d53d321b69cde4464454db94007295b98b3fb4\": container with ID starting with fade27f70677cd84512a6f4825d53d321b69cde4464454db94007295b98b3fb4 not found: ID does not exist" containerID="fade27f70677cd84512a6f4825d53d321b69cde4464454db94007295b98b3fb4" Oct 29 23:41:04.107018 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:04.106962 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fade27f70677cd84512a6f4825d53d321b69cde4464454db94007295b98b3fb4"} err="failed to get container status \"fade27f70677cd84512a6f4825d53d321b69cde4464454db94007295b98b3fb4\": rpc error: code = NotFound desc = could not find container \"fade27f70677cd84512a6f4825d53d321b69cde4464454db94007295b98b3fb4\": container with ID starting with fade27f70677cd84512a6f4825d53d321b69cde4464454db94007295b98b3fb4 not found: ID does not exist" Oct 29 23:41:04.208790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:04.208751 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f2943d7-12ce-4123-a75b-bbbd49a823a5-kube-api-access-4srhw\" (UniqueName: \"kubernetes.io/projected/2f2943d7-12ce-4123-a75b-bbbd49a823a5-kube-api-access-4srhw\") pod \"2f2943d7-12ce-4123-a75b-bbbd49a823a5\" (UID: \"2f2943d7-12ce-4123-a75b-bbbd49a823a5\") " Oct 29 23:41:04.209001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:04.208859 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f2943d7-12ce-4123-a75b-bbbd49a823a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f2943d7-12ce-4123-a75b-bbbd49a823a5-plugins-config-volume\") pod \"2f2943d7-12ce-4123-a75b-bbbd49a823a5\" (UID: \"2f2943d7-12ce-4123-a75b-bbbd49a823a5\") " Oct 29 23:41:04.209245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:04.209222 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2943d7-12ce-4123-a75b-bbbd49a823a5-plugins-config-volume" pod "2f2943d7-12ce-4123-a75b-bbbd49a823a5" (UID: "2f2943d7-12ce-4123-a75b-bbbd49a823a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:41:04.209879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:04.209858 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f2943d7-12ce-4123-a75b-bbbd49a823a5-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:41:04.211494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:04.211469 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2943d7-12ce-4123-a75b-bbbd49a823a5-kube-api-access-4srhw" pod "2f2943d7-12ce-4123-a75b-bbbd49a823a5" (UID: "2f2943d7-12ce-4123-a75b-bbbd49a823a5"). InnerVolumeSpecName "kube-api-access-4srhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:41:04.312058 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:04.311973 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4srhw\" (UniqueName: \"kubernetes.io/projected/2f2943d7-12ce-4123-a75b-bbbd49a823a5-kube-api-access-4srhw\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:41:04.443321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:04.443289 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w"] Oct 29 23:41:04.480165 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:04.480128 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9965bf-96cbc76f5-mnr5w"] Oct 29 23:41:05.554257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:05.546754 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f2943d7-12ce-4123-a75b-bbbd49a823a5" path="/var/lib/kubelet/pods/2f2943d7-12ce-4123-a75b-bbbd49a823a5/volumes" Oct 29 23:41:08.460241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:08.460209 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7"] Oct 29 23:41:08.461768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:08.461741 5057 cpu_manager.go:414] "containerMap: removing container" podUID="2f2943d7-12ce-4123-a75b-bbbd49a823a5" containerName="payload-pre-processing" Oct 29 23:41:08.461865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:08.461770 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2943d7-12ce-4123-a75b-bbbd49a823a5" containerName="payload-pre-processing" Oct 29 23:41:08.462044 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:08.462029 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2943d7-12ce-4123-a75b-bbbd49a823a5" containerName="payload-pre-processing" Oct 29 23:41:08.466505 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:08.466478 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7" Oct 29 23:41:08.486328 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:08.486303 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7"] Oct 29 23:41:08.492636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:08.492608 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d20345\"" Oct 29 23:41:08.493985 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:08.492551 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d20345-dockercfg-zf6ql\"" Oct 29 23:41:08.707210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:08.693643 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed74f944-49b5-453e-8ccb-a46b846e6109-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7\" (UID: \"ed74f944-49b5-453e-8ccb-a46b846e6109\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7" Oct 29 23:41:08.707210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:08.693968 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq7j9\" (UniqueName: \"kubernetes.io/projected/ed74f944-49b5-453e-8ccb-a46b846e6109-kube-api-access-nq7j9\") pod \"payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7\" (UID: \"ed74f944-49b5-453e-8ccb-a46b846e6109\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7" Oct 29 23:41:08.877260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:08.863839 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq7j9\" (UniqueName: \"kubernetes.io/projected/ed74f944-49b5-453e-8ccb-a46b846e6109-kube-api-access-nq7j9\") pod \"payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7\" (UID: \"ed74f944-49b5-453e-8ccb-a46b846e6109\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7" Oct 29 23:41:08.877260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:08.864361 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed74f944-49b5-453e-8ccb-a46b846e6109-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7\" (UID: \"ed74f944-49b5-453e-8ccb-a46b846e6109\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7" Oct 29 23:41:08.877260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:08.865651 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed74f944-49b5-453e-8ccb-a46b846e6109-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7\" (UID: \"ed74f944-49b5-453e-8ccb-a46b846e6109\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7" Oct 29 23:41:08.943270 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:08.943159 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq7j9\" (UniqueName: \"kubernetes.io/projected/ed74f944-49b5-453e-8ccb-a46b846e6109-kube-api-access-nq7j9\") pod \"payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7\" (UID: \"ed74f944-49b5-453e-8ccb-a46b846e6109\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7" Oct 29 23:41:09.083356 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:09.083319 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7" Oct 29 23:41:09.428355 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:41:09.428325 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded74f944_49b5_453e_8ccb_a46b846e6109.slice/crio-9cd4b8325306b307595bfabb251ce20ed92208da5f4c00658144136051e4365a WatchSource:0}: Error finding container 9cd4b8325306b307595bfabb251ce20ed92208da5f4c00658144136051e4365a: Status 404 returned error can't find the container with id 9cd4b8325306b307595bfabb251ce20ed92208da5f4c00658144136051e4365a Oct 29 23:41:09.431015 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:09.430986 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7"] Oct 29 23:41:10.170519 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:10.170472 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7" event={"ID":"ed74f944-49b5-453e-8ccb-a46b846e6109","Type":"ContainerStarted","Data":"869d4a164bc9a8532f5c45f9024248fbfd5329fcae05fa2a6721e57c88cb2bf8"} Oct 29 23:41:10.170992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:10.170534 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7" event={"ID":"ed74f944-49b5-453e-8ccb-a46b846e6109","Type":"ContainerStarted","Data":"9cd4b8325306b307595bfabb251ce20ed92208da5f4c00658144136051e4365a"} Oct 29 23:41:10.170992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:10.170571 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7" Oct 29 23:41:10.215719 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:10.215670 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7" podStartSLOduration=2.215656612 podStartE2EDuration="2.215656612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:41:10.211227356 +0000 UTC m=+3516.862725887" watchObservedRunningTime="2026-10-29 23:41:10.215656612 +0000 UTC m=+3516.867155143" Oct 29 23:41:21.187670 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:21.187631 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7" Oct 29 23:41:33.917708 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:41:33.917669 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:41:58.921658 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:58.921621 5057 generic.go:356] "Generic (PLEG): container finished" podID="a28590ff-f920-4993-b5f2-d8afb67b3685" containerID="3d2dadd12f15bedf8f8b5865e19c4f94f020298f717606fa2fb16a7a4289ce8c" exitCode=0 Oct 29 23:41:58.922235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:58.921911 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" event={"ID":"a28590ff-f920-4993-b5f2-d8afb67b3685","Type":"ContainerDied","Data":"3d2dadd12f15bedf8f8b5865e19c4f94f020298f717606fa2fb16a7a4289ce8c"} Oct 29 23:41:58.922483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:58.922463 5057 scope.go:122] "RemoveContainer" containerID="3d2dadd12f15bedf8f8b5865e19c4f94f020298f717606fa2fb16a7a4289ce8c" Oct 29 23:41:58.925775 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:58.925733 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:41:58.925882 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:58.925836 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:41:58.925927 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:58.925889 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:41:59.935440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:41:59.935401 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" event={"ID":"a28590ff-f920-4993-b5f2-d8afb67b3685","Type":"ContainerStarted","Data":"efa1d67631aad66d2812bd3bfb7c2c4f3a989a84b1b89c59701e81985b7cfd04"} Oct 29 23:42:00.054212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:42:00.051280 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:42:00.066216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:42:00.063760 5057 patch_prober.go:29] interesting pod/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Oct 29 23:42:00.066216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:42:00.063832 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" podUID="a28590ff-f920-4993-b5f2-d8afb67b3685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Oct 29 23:42:01.054214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:42:01.050608 5057 patch_prober.go:29] interesting pod/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Oct 29 23:42:01.054214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:42:01.050689 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" podUID="a28590ff-f920-4993-b5f2-d8afb67b3685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Oct 29 23:42:02.050212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:42:02.049545 5057 patch_prober.go:29] interesting pod/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Oct 29 23:42:02.050212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:42:02.049611 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" podUID="a28590ff-f920-4993-b5f2-d8afb67b3685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Oct 29 23:42:03.051210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:42:03.049567 5057 patch_prober.go:29] interesting pod/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Oct 29 23:42:03.051210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:42:03.049635 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" podUID="a28590ff-f920-4993-b5f2-d8afb67b3685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Oct 29 23:42:03.917160 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:42:03.917118 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:42:04.053437 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:42:04.050443 5057 patch_prober.go:29] interesting pod/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Oct 29 23:42:04.053961 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:42:04.053477 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" podUID="a28590ff-f920-4993-b5f2-d8afb67b3685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Oct 29 23:42:05.049675 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:42:05.049633 5057 patch_prober.go:29] interesting pod/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Oct 29 23:42:05.049897 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:42:05.049715 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" podUID="a28590ff-f920-4993-b5f2-d8afb67b3685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Oct 29 23:42:05.442327 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:42:05.442286 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:42:06.183428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:42:06.183365 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:42:06.185121 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:42:06.185092 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:42:33.916754 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:42:33.916714 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:42:35.284569 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:42:35.284535 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:42:35.300432 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:42:35.300406 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:42:41.776310 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:42:41.776254 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:43:03.916913 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:43:03.916870 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:43:20.891204 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:20.890957 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt"] Oct 29 23:43:30.510966 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:30.510927 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv"] Oct 29 23:43:33.915952 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:43:33.915918 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:43:35.511272 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:35.511220 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv" podUID="664aecb9-c880-4cb4-806a-293858b56120" containerName="payload-pre-processing" containerID="cri-o://8da79cdf2cf1e6cba6707f76965a0da907589df30229ba43694930b11555e70c" gracePeriod=25 Oct 29 23:43:35.815970 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:35.815938 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv" Oct 29 23:43:35.937523 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:35.937481 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/664aecb9-c880-4cb4-806a-293858b56120-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664aecb9-c880-4cb4-806a-293858b56120-plugins-config-volume\") pod \"664aecb9-c880-4cb4-806a-293858b56120\" (UID: \"664aecb9-c880-4cb4-806a-293858b56120\") " Oct 29 23:43:35.937915 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:35.937893 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/664aecb9-c880-4cb4-806a-293858b56120-kube-api-access-8hms5\" (UniqueName: \"kubernetes.io/projected/664aecb9-c880-4cb4-806a-293858b56120-kube-api-access-8hms5\") pod \"664aecb9-c880-4cb4-806a-293858b56120\" (UID: \"664aecb9-c880-4cb4-806a-293858b56120\") " Oct 29 23:43:35.939080 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:35.939042 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/664aecb9-c880-4cb4-806a-293858b56120-plugins-config-volume" pod "664aecb9-c880-4cb4-806a-293858b56120" (UID: "664aecb9-c880-4cb4-806a-293858b56120"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:43:35.940504 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:35.940466 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/664aecb9-c880-4cb4-806a-293858b56120-kube-api-access-8hms5" pod "664aecb9-c880-4cb4-806a-293858b56120" (UID: "664aecb9-c880-4cb4-806a-293858b56120"). InnerVolumeSpecName "kube-api-access-8hms5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:43:35.944124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:35.943966 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" podUID="9033febc-c7ee-4884-9145-c27229f9f5dd" containerName="main" containerID="cri-o://c8c02cc5919139e1d257fc16fb1d7cf79f573f42f43d57d1a2ad258ed94f0b8e" gracePeriod=45 Oct 29 23:43:35.972604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:35.972566 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" podUID="9033febc-c7ee-4884-9145-c27229f9f5dd" containerName="main" probeResult="failure" output="Get \"https://10.132.1.40:8000/ready\": dial tcp 10.132.1.40:8000: connect: connection refused" Oct 29 23:43:36.040386 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.040312 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hms5\" (UniqueName: \"kubernetes.io/projected/664aecb9-c880-4cb4-806a-293858b56120-kube-api-access-8hms5\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:43:36.040386 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.040340 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664aecb9-c880-4cb4-806a-293858b56120-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:43:36.184708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.184671 5057 generic.go:356] "Generic (PLEG): container finished" podID="664aecb9-c880-4cb4-806a-293858b56120" containerID="8da79cdf2cf1e6cba6707f76965a0da907589df30229ba43694930b11555e70c" exitCode=0 Oct 29 23:43:36.185034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.185011 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv" Oct 29 23:43:36.185164 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.185062 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv" event={"ID":"664aecb9-c880-4cb4-806a-293858b56120","Type":"ContainerDied","Data":"8da79cdf2cf1e6cba6707f76965a0da907589df30229ba43694930b11555e70c"} Oct 29 23:43:36.185164 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.185135 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv" event={"ID":"664aecb9-c880-4cb4-806a-293858b56120","Type":"ContainerDied","Data":"8d3af56d863544b32f25bc5ad8cd4410b33355e6c546f6c5f65200848bf5e11b"} Oct 29 23:43:36.185164 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.185163 5057 scope.go:122] "RemoveContainer" containerID="8da79cdf2cf1e6cba6707f76965a0da907589df30229ba43694930b11555e70c" Oct 29 23:43:36.189892 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.189784 5057 generic.go:356] "Generic (PLEG): container finished" podID="9033febc-c7ee-4884-9145-c27229f9f5dd" containerID="c8c02cc5919139e1d257fc16fb1d7cf79f573f42f43d57d1a2ad258ed94f0b8e" exitCode=0 Oct 29 23:43:36.189892 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.189845 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" event={"ID":"9033febc-c7ee-4884-9145-c27229f9f5dd","Type":"ContainerDied","Data":"c8c02cc5919139e1d257fc16fb1d7cf79f573f42f43d57d1a2ad258ed94f0b8e"} Oct 29 23:43:36.204309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.204286 5057 scope.go:122] "RemoveContainer" containerID="8da79cdf2cf1e6cba6707f76965a0da907589df30229ba43694930b11555e70c" Oct 29 23:43:36.204654 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:43:36.204630 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8da79cdf2cf1e6cba6707f76965a0da907589df30229ba43694930b11555e70c\": container with ID starting with 8da79cdf2cf1e6cba6707f76965a0da907589df30229ba43694930b11555e70c not found: ID does not exist" containerID="8da79cdf2cf1e6cba6707f76965a0da907589df30229ba43694930b11555e70c" Oct 29 23:43:36.204767 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.204664 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8da79cdf2cf1e6cba6707f76965a0da907589df30229ba43694930b11555e70c"} err="failed to get container status \"8da79cdf2cf1e6cba6707f76965a0da907589df30229ba43694930b11555e70c\": rpc error: code = NotFound desc = could not find container \"8da79cdf2cf1e6cba6707f76965a0da907589df30229ba43694930b11555e70c\": container with ID starting with 8da79cdf2cf1e6cba6707f76965a0da907589df30229ba43694930b11555e70c not found: ID does not exist" Oct 29 23:43:36.241715 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.241592 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv"] Oct 29 23:43:36.251937 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.251903 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a8ab3c-cddcf794c-8bgqv"] Oct 29 23:43:36.257477 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.257453 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:43:36.449083 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.449049 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-model-cache\") pod \"9033febc-c7ee-4884-9145-c27229f9f5dd\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " Oct 29 23:43:36.449299 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.449270 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-dshm\") pod \"9033febc-c7ee-4884-9145-c27229f9f5dd\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " Oct 29 23:43:36.449409 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.449381 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-model-cache" pod "9033febc-c7ee-4884-9145-c27229f9f5dd" (UID: "9033febc-c7ee-4884-9145-c27229f9f5dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:36.449471 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.449403 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-tmp-dir\") pod \"9033febc-c7ee-4884-9145-c27229f9f5dd\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " Oct 29 23:43:36.449533 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.449474 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9033febc-c7ee-4884-9145-c27229f9f5dd-tls-certs\" (UniqueName: \"kubernetes.io/secret/9033febc-c7ee-4884-9145-c27229f9f5dd-tls-certs\") pod \"9033febc-c7ee-4884-9145-c27229f9f5dd\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " Oct 29 23:43:36.449593 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.449545 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9033febc-c7ee-4884-9145-c27229f9f5dd-kube-api-access-kpxbn\" (UniqueName: \"kubernetes.io/projected/9033febc-c7ee-4884-9145-c27229f9f5dd-kube-api-access-kpxbn\") pod \"9033febc-c7ee-4884-9145-c27229f9f5dd\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " Oct 29 23:43:36.449713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.449695 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-home\" (UniqueName: \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-home\") pod \"9033febc-c7ee-4884-9145-c27229f9f5dd\" (UID: \"9033febc-c7ee-4884-9145-c27229f9f5dd\") " Oct 29 23:43:36.449870 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.449696 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-tmp-dir" pod "9033febc-c7ee-4884-9145-c27229f9f5dd" (UID: "9033febc-c7ee-4884-9145-c27229f9f5dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:36.449934 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.449882 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-home" pod "9033febc-c7ee-4884-9145-c27229f9f5dd" (UID: "9033febc-c7ee-4884-9145-c27229f9f5dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:36.450621 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.450580 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-tmp-dir\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:43:36.450749 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.450626 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-home\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:43:36.450749 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.450641 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-model-cache\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:43:36.452003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.451980 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9033febc-c7ee-4884-9145-c27229f9f5dd-tls-certs" pod "9033febc-c7ee-4884-9145-c27229f9f5dd" (UID: "9033febc-c7ee-4884-9145-c27229f9f5dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:43:36.452109 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.452051 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-dshm" pod "9033febc-c7ee-4884-9145-c27229f9f5dd" (UID: "9033febc-c7ee-4884-9145-c27229f9f5dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:36.452235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.452172 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9033febc-c7ee-4884-9145-c27229f9f5dd-kube-api-access-kpxbn" pod "9033febc-c7ee-4884-9145-c27229f9f5dd" (UID: "9033febc-c7ee-4884-9145-c27229f9f5dd"). InnerVolumeSpecName "kube-api-access-kpxbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:43:36.552717 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.552681 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9033febc-c7ee-4884-9145-c27229f9f5dd-tls-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:43:36.553060 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.552721 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpxbn\" (UniqueName: \"kubernetes.io/projected/9033febc-c7ee-4884-9145-c27229f9f5dd-kube-api-access-kpxbn\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:43:36.553060 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:36.552736 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9033febc-c7ee-4884-9145-c27229f9f5dd-dshm\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:43:37.200761 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:37.200726 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" Oct 29 23:43:37.200970 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:37.200758 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt" event={"ID":"9033febc-c7ee-4884-9145-c27229f9f5dd","Type":"ContainerDied","Data":"e9db943eb609dc71c35cdc0d476f7bf22719119d9e36758f12f1cdb900b2300a"} Oct 29 23:43:37.200970 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:37.200813 5057 scope.go:122] "RemoveContainer" containerID="c8c02cc5919139e1d257fc16fb1d7cf79f573f42f43d57d1a2ad258ed94f0b8e" Oct 29 23:43:37.255872 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:37.255836 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt"] Oct 29 23:43:37.260713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:37.260685 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-5cd0f2/test-model-5cd0f2-kserve-5646c5b5c9-dx6pt"] Oct 29 23:43:37.484173 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:37.484098 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="664aecb9-c880-4cb4-806a-293858b56120" path="/var/lib/kubelet/pods/664aecb9-c880-4cb4-806a-293858b56120/volumes" Oct 29 23:43:37.484625 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:37.484609 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9033febc-c7ee-4884-9145-c27229f9f5dd" path="/var/lib/kubelet/pods/9033febc-c7ee-4884-9145-c27229f9f5dd/volumes" Oct 29 23:43:47.471752 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:47.471713 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h"] Oct 29 23:43:47.472271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:47.471985 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" podUID="0d81f061-78fe-4a4a-aba1-11cc32a218c6" containerName="istio-proxy" containerID="cri-o://a2557cbb2efa64eb45f1b7c00ce76c5ed038779c1a6d824ffab2d7e8cd9e9e38" gracePeriod=30 Oct 29 23:43:52.974024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:52.973986 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:43:53.128827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.128739 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-workload-certs\") pod \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " Oct 29 23:43:53.129006 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.128869 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-data\") pod \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " Oct 29 23:43:53.129006 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.128918 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-podinfo\") pod \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " Oct 29 23:43:53.129129 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.129027 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-envoy\") pod \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " Oct 29 23:43:53.129129 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.129064 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istiod-ca-cert\") pod \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " Oct 29 23:43:53.129293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.129203 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-token\" (UniqueName: \"kubernetes.io/projected/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-token\") pod \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " Oct 29 23:43:53.129293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.129260 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-credential-socket\") pod \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " Oct 29 23:43:53.129404 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.129350 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d81f061-78fe-4a4a-aba1-11cc32a218c6-kube-api-access-dvzb5\" (UniqueName: \"kubernetes.io/projected/0d81f061-78fe-4a4a-aba1-11cc32a218c6-kube-api-access-dvzb5\") pod \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " Oct 29 23:43:53.129465 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.129411 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-workload-socket\") pod \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\" (UID: \"0d81f061-78fe-4a4a-aba1-11cc32a218c6\") " Oct 29 23:43:53.130374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.130061 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istiod-ca-cert" pod "0d81f061-78fe-4a4a-aba1-11cc32a218c6" (UID: "0d81f061-78fe-4a4a-aba1-11cc32a218c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:43:53.130374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.130278 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-credential-socket" pod "0d81f061-78fe-4a4a-aba1-11cc32a218c6" (UID: "0d81f061-78fe-4a4a-aba1-11cc32a218c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:53.130374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.130367 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:43:53.130707 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.130556 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-workload-certs" pod "0d81f061-78fe-4a4a-aba1-11cc32a218c6" (UID: "0d81f061-78fe-4a4a-aba1-11cc32a218c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:53.130707 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.130603 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-data" pod "0d81f061-78fe-4a4a-aba1-11cc32a218c6" (UID: "0d81f061-78fe-4a4a-aba1-11cc32a218c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:53.130969 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.130943 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-workload-socket" pod "0d81f061-78fe-4a4a-aba1-11cc32a218c6" (UID: "0d81f061-78fe-4a4a-aba1-11cc32a218c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:53.131965 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.131941 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-token" pod "0d81f061-78fe-4a4a-aba1-11cc32a218c6" (UID: "0d81f061-78fe-4a4a-aba1-11cc32a218c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:43:53.132226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.132163 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-podinfo" pod "0d81f061-78fe-4a4a-aba1-11cc32a218c6" (UID: "0d81f061-78fe-4a4a-aba1-11cc32a218c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:43:53.133105 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.133080 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-envoy" pod "0d81f061-78fe-4a4a-aba1-11cc32a218c6" (UID: "0d81f061-78fe-4a4a-aba1-11cc32a218c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:53.133981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.133955 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d81f061-78fe-4a4a-aba1-11cc32a218c6-kube-api-access-dvzb5" pod "0d81f061-78fe-4a4a-aba1-11cc32a218c6" (UID: "0d81f061-78fe-4a4a-aba1-11cc32a218c6"). InnerVolumeSpecName "kube-api-access-dvzb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:43:53.231562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.231526 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:43:53.231562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.231555 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:43:53.231562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.231564 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:43:53.231562 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.231572 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d81f061-78fe-4a4a-aba1-11cc32a218c6-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:43:53.231840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.231581 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:43:53.231840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.231590 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvzb5\" (UniqueName: \"kubernetes.io/projected/0d81f061-78fe-4a4a-aba1-11cc32a218c6-kube-api-access-dvzb5\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:43:53.231840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.231600 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:43:53.231840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.231609 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d81f061-78fe-4a4a-aba1-11cc32a218c6-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:43:53.549729 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.549692 5057 generic.go:356] "Generic (PLEG): container finished" podID="0d81f061-78fe-4a4a-aba1-11cc32a218c6" containerID="a2557cbb2efa64eb45f1b7c00ce76c5ed038779c1a6d824ffab2d7e8cd9e9e38" exitCode=0 Oct 29 23:43:53.549922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.549759 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" Oct 29 23:43:53.549922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.549775 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" event={"ID":"0d81f061-78fe-4a4a-aba1-11cc32a218c6","Type":"ContainerDied","Data":"a2557cbb2efa64eb45f1b7c00ce76c5ed038779c1a6d824ffab2d7e8cd9e9e38"} Oct 29 23:43:53.549922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.549834 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h" event={"ID":"0d81f061-78fe-4a4a-aba1-11cc32a218c6","Type":"ContainerDied","Data":"149d72144527ae5d4365e3ec77d33675020c8228c3d8342e2acb840ccbc11052"} Oct 29 23:43:53.549922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.549858 5057 scope.go:122] "RemoveContainer" containerID="a2557cbb2efa64eb45f1b7c00ce76c5ed038779c1a6d824ffab2d7e8cd9e9e38" Oct 29 23:43:53.562157 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.562138 5057 scope.go:122] "RemoveContainer" containerID="41ae0ebf6b52de4ab6e7ddd0f3107404a7b0a677997eec68765b437e49f80efe" Oct 29 23:43:53.575066 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.575047 5057 scope.go:122] "RemoveContainer" containerID="a2557cbb2efa64eb45f1b7c00ce76c5ed038779c1a6d824ffab2d7e8cd9e9e38" Oct 29 23:43:53.575387 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:43:53.575366 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2557cbb2efa64eb45f1b7c00ce76c5ed038779c1a6d824ffab2d7e8cd9e9e38\": container with ID starting with a2557cbb2efa64eb45f1b7c00ce76c5ed038779c1a6d824ffab2d7e8cd9e9e38 not found: ID does not exist" containerID="a2557cbb2efa64eb45f1b7c00ce76c5ed038779c1a6d824ffab2d7e8cd9e9e38" Oct 29 23:43:53.575462 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.575396 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2557cbb2efa64eb45f1b7c00ce76c5ed038779c1a6d824ffab2d7e8cd9e9e38"} err="failed to get container status \"a2557cbb2efa64eb45f1b7c00ce76c5ed038779c1a6d824ffab2d7e8cd9e9e38\": rpc error: code = NotFound desc = could not find container \"a2557cbb2efa64eb45f1b7c00ce76c5ed038779c1a6d824ffab2d7e8cd9e9e38\": container with ID starting with a2557cbb2efa64eb45f1b7c00ce76c5ed038779c1a6d824ffab2d7e8cd9e9e38 not found: ID does not exist" Oct 29 23:43:53.575462 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.575413 5057 scope.go:122] "RemoveContainer" containerID="41ae0ebf6b52de4ab6e7ddd0f3107404a7b0a677997eec68765b437e49f80efe" Oct 29 23:43:53.575663 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:43:53.575646 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41ae0ebf6b52de4ab6e7ddd0f3107404a7b0a677997eec68765b437e49f80efe\": container with ID starting with 41ae0ebf6b52de4ab6e7ddd0f3107404a7b0a677997eec68765b437e49f80efe not found: ID does not exist" containerID="41ae0ebf6b52de4ab6e7ddd0f3107404a7b0a677997eec68765b437e49f80efe" Oct 29 23:43:53.575705 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.575669 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41ae0ebf6b52de4ab6e7ddd0f3107404a7b0a677997eec68765b437e49f80efe"} err="failed to get container status \"41ae0ebf6b52de4ab6e7ddd0f3107404a7b0a677997eec68765b437e49f80efe\": rpc error: code = NotFound desc = could not find container \"41ae0ebf6b52de4ab6e7ddd0f3107404a7b0a677997eec68765b437e49f80efe\": container with ID starting with 41ae0ebf6b52de4ab6e7ddd0f3107404a7b0a677997eec68765b437e49f80efe not found: ID does not exist" Oct 29 23:43:53.618531 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.618498 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h"] Oct 29 23:43:53.630298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:53.630271 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-5gj2h"] Oct 29 23:43:55.485440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:55.485398 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d81f061-78fe-4a4a-aba1-11cc32a218c6" path="/var/lib/kubelet/pods/0d81f061-78fe-4a4a-aba1-11cc32a218c6/volumes" Oct 29 23:43:57.495344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:43:57.495308 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d"] Oct 29 23:44:02.496259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:02.496220 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d" podUID="d0f270e1-9bac-4a4c-a472-4d4b07fa7894" containerName="payload-pre-processing" containerID="cri-o://7154e59da8308cb34079990640813e5b9ea588618ff78f26e51467078dfd4e56" gracePeriod=25 Oct 29 23:44:03.915235 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:44:03.915165 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:44:06.635592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:06.635550 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d" podUID="d0f270e1-9bac-4a4c-a472-4d4b07fa7894" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:44:09.492107 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:09.492069 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr"] Oct 29 23:44:09.492627 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:09.492328 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" podUID="634af89c-ecba-484c-8843-5154b3676abb" containerName="maas-api" containerID="cri-o://c1edcea327342bfb5395ed68df45d0c37cdeca7657c323c527e3f631d3d2266d" gracePeriod=30 Oct 29 23:44:10.235741 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:10.235698 5057 generic.go:356] "Generic (PLEG): container finished" podID="634af89c-ecba-484c-8843-5154b3676abb" containerID="c1edcea327342bfb5395ed68df45d0c37cdeca7657c323c527e3f631d3d2266d" exitCode=0 Oct 29 23:44:10.236223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:10.236172 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" event={"ID":"634af89c-ecba-484c-8843-5154b3676abb","Type":"ContainerDied","Data":"c1edcea327342bfb5395ed68df45d0c37cdeca7657c323c527e3f631d3d2266d"} Oct 29 23:44:10.366732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:10.366647 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" Oct 29 23:44:10.469693 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:10.469659 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/634af89c-ecba-484c-8843-5154b3676abb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/634af89c-ecba-484c-8843-5154b3676abb-maas-api-tls\") pod \"634af89c-ecba-484c-8843-5154b3676abb\" (UID: \"634af89c-ecba-484c-8843-5154b3676abb\") " Oct 29 23:44:10.470179 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:10.470153 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/634af89c-ecba-484c-8843-5154b3676abb-kube-api-access-bwgw2\" (UniqueName: \"kubernetes.io/projected/634af89c-ecba-484c-8843-5154b3676abb-kube-api-access-bwgw2\") pod \"634af89c-ecba-484c-8843-5154b3676abb\" (UID: \"634af89c-ecba-484c-8843-5154b3676abb\") " Oct 29 23:44:10.477382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:10.477359 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/634af89c-ecba-484c-8843-5154b3676abb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/634af89c-ecba-484c-8843-5154b3676abb-openshift-service-ca\") pod \"634af89c-ecba-484c-8843-5154b3676abb\" (UID: \"634af89c-ecba-484c-8843-5154b3676abb\") " Oct 29 23:44:10.489476 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:10.477850 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634af89c-ecba-484c-8843-5154b3676abb-openshift-service-ca" pod "634af89c-ecba-484c-8843-5154b3676abb" (UID: "634af89c-ecba-484c-8843-5154b3676abb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:44:10.494781 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:10.494737 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/634af89c-ecba-484c-8843-5154b3676abb-maas-api-tls" pod "634af89c-ecba-484c-8843-5154b3676abb" (UID: "634af89c-ecba-484c-8843-5154b3676abb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:44:10.496012 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:10.495983 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/634af89c-ecba-484c-8843-5154b3676abb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/634af89c-ecba-484c-8843-5154b3676abb-metrics-certs\") pod \"634af89c-ecba-484c-8843-5154b3676abb\" (UID: \"634af89c-ecba-484c-8843-5154b3676abb\") " Oct 29 23:44:10.496875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:10.496818 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/634af89c-ecba-484c-8843-5154b3676abb-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:10.496875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:10.496847 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/634af89c-ecba-484c-8843-5154b3676abb-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:10.497028 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:10.497010 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/634af89c-ecba-484c-8843-5154b3676abb-kube-api-access-bwgw2" pod "634af89c-ecba-484c-8843-5154b3676abb" (UID: "634af89c-ecba-484c-8843-5154b3676abb"). InnerVolumeSpecName "kube-api-access-bwgw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:10.514231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:10.514130 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/634af89c-ecba-484c-8843-5154b3676abb-metrics-certs" pod "634af89c-ecba-484c-8843-5154b3676abb" (UID: "634af89c-ecba-484c-8843-5154b3676abb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:44:10.603150 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:10.602553 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwgw2\" (UniqueName: \"kubernetes.io/projected/634af89c-ecba-484c-8843-5154b3676abb-kube-api-access-bwgw2\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:10.603150 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:10.602997 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/634af89c-ecba-484c-8843-5154b3676abb-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:11.249335 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:11.249307 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" Oct 29 23:44:11.249559 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:11.249348 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" event={"ID":"634af89c-ecba-484c-8843-5154b3676abb","Type":"ContainerDied","Data":"5e062f1a702f754078fc14c291456bf8aee9a48aa7952f360f9cbe127986ba1e"} Oct 29 23:44:11.249559 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:11.249422 5057 scope.go:122] "RemoveContainer" containerID="c1edcea327342bfb5395ed68df45d0c37cdeca7657c323c527e3f631d3d2266d" Oct 29 23:44:11.304890 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:11.304862 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr"] Oct 29 23:44:11.315775 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:11.315745 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr"] Oct 29 23:44:11.484757 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:11.484726 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="634af89c-ecba-484c-8843-5154b3676abb" path="/var/lib/kubelet/pods/634af89c-ecba-484c-8843-5154b3676abb/volumes" Oct 29 23:44:12.963017 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:12.962962 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a8ab3c-848d67788b-mm4gr" podUID="634af89c-ecba-484c-8843-5154b3676abb" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.1.45:8443/health\": context deadline exceeded" Oct 29 23:44:16.635512 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:16.635467 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d" podUID="d0f270e1-9bac-4a4c-a472-4d4b07fa7894" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:44:18.128647 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.128613 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd"] Oct 29 23:44:18.129617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.129568 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9033febc-c7ee-4884-9145-c27229f9f5dd" containerName="main" Oct 29 23:44:18.129617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.129586 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9033febc-c7ee-4884-9145-c27229f9f5dd" containerName="main" Oct 29 23:44:18.129794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.129624 5057 cpu_manager.go:414] "containerMap: removing container" podUID="664aecb9-c880-4cb4-806a-293858b56120" containerName="payload-pre-processing" Oct 29 23:44:18.129794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.129632 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="664aecb9-c880-4cb4-806a-293858b56120" containerName="payload-pre-processing" Oct 29 23:44:18.129794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.129658 5057 cpu_manager.go:414] "containerMap: removing container" podUID="0d81f061-78fe-4a4a-aba1-11cc32a218c6" containerName="istio-proxy" Oct 29 23:44:18.129794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.129665 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d81f061-78fe-4a4a-aba1-11cc32a218c6" containerName="istio-proxy" Oct 29 23:44:18.129794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.129688 5057 cpu_manager.go:414] "containerMap: removing container" podUID="0d81f061-78fe-4a4a-aba1-11cc32a218c6" containerName="istio-proxy" Oct 29 23:44:18.129794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.129694 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d81f061-78fe-4a4a-aba1-11cc32a218c6" containerName="istio-proxy" Oct 29 23:44:18.129794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.129721 5057 cpu_manager.go:414] "containerMap: removing container" podUID="634af89c-ecba-484c-8843-5154b3676abb" containerName="maas-api" Oct 29 23:44:18.129794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.129726 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="634af89c-ecba-484c-8843-5154b3676abb" containerName="maas-api" Oct 29 23:44:18.130086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.129944 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="664aecb9-c880-4cb4-806a-293858b56120" containerName="payload-pre-processing" Oct 29 23:44:18.130086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.129963 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d81f061-78fe-4a4a-aba1-11cc32a218c6" containerName="istio-proxy" Oct 29 23:44:18.130086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.129972 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d81f061-78fe-4a4a-aba1-11cc32a218c6" containerName="istio-proxy" Oct 29 23:44:18.130086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.129985 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="634af89c-ecba-484c-8843-5154b3676abb" containerName="maas-api" Oct 29 23:44:18.130086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.129998 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9033febc-c7ee-4884-9145-c27229f9f5dd" containerName="main" Oct 29 23:44:18.135763 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.135735 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.150678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.150655 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b7a567-openshift-default-dockercfg-pms6g\"" Oct 29 23:44:18.169574 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.169547 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd"] Oct 29 23:44:18.210788 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.210745 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-podinfo\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.210980 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.210918 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-token\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.210980 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.210955 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-envoy\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.211075 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.211062 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-workload-socket\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.211113 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.211086 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-credential-socket\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.211163 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.211122 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7trbp\" (UniqueName: \"kubernetes.io/projected/72f6fe5b-dae2-4624-b4f4-bb329f370b87-kube-api-access-7trbp\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.211301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.211280 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-workload-certs\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.211415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.211397 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istiod-ca-cert\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.211494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.211478 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-data\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.313458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.313417 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-workload-socket\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.313458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.313466 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-credential-socket\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.313732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.313508 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7trbp\" (UniqueName: \"kubernetes.io/projected/72f6fe5b-dae2-4624-b4f4-bb329f370b87-kube-api-access-7trbp\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.313732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.313597 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-workload-certs\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.313732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.313710 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istiod-ca-cert\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.313909 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.313785 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-data\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.313909 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.313831 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-podinfo\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.313909 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.313872 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-workload-socket\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.314089 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.313923 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-credential-socket\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.314089 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.313970 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-workload-certs\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.314089 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.314049 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-token\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.314287 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.314088 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-envoy\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.314625 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.314598 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istiod-ca-cert\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.314918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.314897 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-data\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.316425 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.316405 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-envoy\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.316535 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.316471 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-podinfo\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.354955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.354922 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-token\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.359631 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.359605 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7trbp\" (UniqueName: \"kubernetes.io/projected/72f6fe5b-dae2-4624-b4f4-bb329f370b87-kube-api-access-7trbp\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:18.451228 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:18.451169 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:19.030958 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:19.030923 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd"] Oct 29 23:44:19.036697 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:44:19.036671 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f6fe5b_dae2_4624_b4f4_bb329f370b87.slice/crio-c128c5cb0a95045ad4fad085de230a9e49e3ae27ed087e1c1699ba70e70badea WatchSource:0}: Error finding container c128c5cb0a95045ad4fad085de230a9e49e3ae27ed087e1c1699ba70e70badea: Status 404 returned error can't find the container with id c128c5cb0a95045ad4fad085de230a9e49e3ae27ed087e1c1699ba70e70badea Oct 29 23:44:19.040812 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:19.040763 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:44:19.040915 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:19.040855 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:44:19.040915 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:19.040894 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:44:19.427665 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:19.427624 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" event={"ID":"72f6fe5b-dae2-4624-b4f4-bb329f370b87","Type":"ContainerStarted","Data":"4d58f54a6ab5734974a78e6a15eb75dda3fb83c69148dec68f7ea241db5a3f44"} Oct 29 23:44:19.428142 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:19.427693 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" event={"ID":"72f6fe5b-dae2-4624-b4f4-bb329f370b87","Type":"ContainerStarted","Data":"c128c5cb0a95045ad4fad085de230a9e49e3ae27ed087e1c1699ba70e70badea"} Oct 29 23:44:19.451545 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:19.451516 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:19.481240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:19.481164 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" podStartSLOduration=1.481146246 podStartE2EDuration="1.481146246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:44:19.476832454 +0000 UTC m=+3706.128330983" watchObservedRunningTime="2026-10-29 23:44:19.481146246 +0000 UTC m=+3706.132644777" Oct 29 23:44:20.466314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:20.466070 5057 patch_prober.go:29] interesting pod/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 29 23:44:20.466314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:20.466128 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" podUID="72f6fe5b-dae2-4624-b4f4-bb329f370b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 29 23:44:21.452677 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:21.452636 5057 patch_prober.go:29] interesting pod/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 29 23:44:21.452894 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:21.452859 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" podUID="72f6fe5b-dae2-4624-b4f4-bb329f370b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 29 23:44:22.464167 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:22.463665 5057 patch_prober.go:29] interesting pod/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 29 23:44:22.464167 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:22.463727 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" podUID="72f6fe5b-dae2-4624-b4f4-bb329f370b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 29 23:44:22.696164 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:22.696118 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq"] Oct 29 23:44:22.697134 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:22.697092 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" podUID="8645590d-04a6-4cff-a65e-281d5cb7f059" containerName="istio-proxy" containerID="cri-o://3fa14b2805b49b0e8faf99ea4ab05b38ab3b8ec105d0be50397a543aa65a8aa4" gracePeriod=30 Oct 29 23:44:23.428343 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:23.428314 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d" Oct 29 23:44:23.458907 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:23.458867 5057 patch_prober.go:29] interesting pod/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 29 23:44:23.459105 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:23.458915 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" podUID="72f6fe5b-dae2-4624-b4f4-bb329f370b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 29 23:44:23.601210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:23.588768 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f270e1-9bac-4a4c-a472-4d4b07fa7894-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0f270e1-9bac-4a4c-a472-4d4b07fa7894-plugins-config-volume\") pod \"d0f270e1-9bac-4a4c-a472-4d4b07fa7894\" (UID: \"d0f270e1-9bac-4a4c-a472-4d4b07fa7894\") " Oct 29 23:44:23.619075 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:23.607865 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f270e1-9bac-4a4c-a472-4d4b07fa7894-kube-api-access-kzjth\" (UniqueName: \"kubernetes.io/projected/d0f270e1-9bac-4a4c-a472-4d4b07fa7894-kube-api-access-kzjth\") pod \"d0f270e1-9bac-4a4c-a472-4d4b07fa7894\" (UID: \"d0f270e1-9bac-4a4c-a472-4d4b07fa7894\") " Oct 29 23:44:23.632209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:23.603513 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f270e1-9bac-4a4c-a472-4d4b07fa7894-plugins-config-volume" pod "d0f270e1-9bac-4a4c-a472-4d4b07fa7894" (UID: "d0f270e1-9bac-4a4c-a472-4d4b07fa7894"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:44:23.670728 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:23.669577 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f270e1-9bac-4a4c-a472-4d4b07fa7894-kube-api-access-kzjth" pod "d0f270e1-9bac-4a4c-a472-4d4b07fa7894" (UID: "d0f270e1-9bac-4a4c-a472-4d4b07fa7894"). InnerVolumeSpecName "kube-api-access-kzjth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:23.673923 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:23.673888 5057 generic.go:356] "Generic (PLEG): container finished" podID="d0f270e1-9bac-4a4c-a472-4d4b07fa7894" containerID="7154e59da8308cb34079990640813e5b9ea588618ff78f26e51467078dfd4e56" exitCode=0 Oct 29 23:44:23.674062 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:23.673938 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d" event={"ID":"d0f270e1-9bac-4a4c-a472-4d4b07fa7894","Type":"ContainerDied","Data":"7154e59da8308cb34079990640813e5b9ea588618ff78f26e51467078dfd4e56"} Oct 29 23:44:23.674062 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:23.673988 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d" event={"ID":"d0f270e1-9bac-4a4c-a472-4d4b07fa7894","Type":"ContainerDied","Data":"4d6a75212032b0728aa782edc2ab5890aeddb74811504bd8ba36bc6631d7f48a"} Oct 29 23:44:23.674062 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:23.674015 5057 scope.go:122] "RemoveContainer" containerID="7154e59da8308cb34079990640813e5b9ea588618ff78f26e51467078dfd4e56" Oct 29 23:44:23.674252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:23.674219 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d" Oct 29 23:44:23.692510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:23.692482 5057 scope.go:122] "RemoveContainer" containerID="7154e59da8308cb34079990640813e5b9ea588618ff78f26e51467078dfd4e56" Oct 29 23:44:23.692897 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:44:23.692876 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7154e59da8308cb34079990640813e5b9ea588618ff78f26e51467078dfd4e56\": container with ID starting with 7154e59da8308cb34079990640813e5b9ea588618ff78f26e51467078dfd4e56 not found: ID does not exist" containerID="7154e59da8308cb34079990640813e5b9ea588618ff78f26e51467078dfd4e56" Oct 29 23:44:23.692972 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:23.692905 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7154e59da8308cb34079990640813e5b9ea588618ff78f26e51467078dfd4e56"} err="failed to get container status \"7154e59da8308cb34079990640813e5b9ea588618ff78f26e51467078dfd4e56\": rpc error: code = NotFound desc = could not find container \"7154e59da8308cb34079990640813e5b9ea588618ff78f26e51467078dfd4e56\": container with ID starting with 7154e59da8308cb34079990640813e5b9ea588618ff78f26e51467078dfd4e56 not found: ID does not exist" Oct 29 23:44:23.750748 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:23.741040 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0f270e1-9bac-4a4c-a472-4d4b07fa7894-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:23.750748 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:23.741081 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzjth\" (UniqueName: \"kubernetes.io/projected/d0f270e1-9bac-4a4c-a472-4d4b07fa7894-kube-api-access-kzjth\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:23.756705 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:23.756672 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d"] Oct 29 23:44:23.761739 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:23.761710 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5cd0f2-6669fdf5c5-hnt9d"] Oct 29 23:44:24.452215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:24.451752 5057 patch_prober.go:29] interesting pod/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 29 23:44:24.452215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:24.451837 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" podUID="72f6fe5b-dae2-4624-b4f4-bb329f370b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 29 23:44:25.457209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:25.457153 5057 patch_prober.go:29] interesting pod/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 29 23:44:25.457747 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:25.457717 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" podUID="72f6fe5b-dae2-4624-b4f4-bb329f370b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 29 23:44:25.489463 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:25.489426 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f270e1-9bac-4a4c-a472-4d4b07fa7894" path="/var/lib/kubelet/pods/d0f270e1-9bac-4a4c-a472-4d4b07fa7894/volumes" Oct 29 23:44:25.502939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:25.502899 5057 patch_prober.go:29] interesting pod/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:44:25.503115 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:25.502962 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" podUID="8645590d-04a6-4cff-a65e-281d5cb7f059" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:44:26.457907 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:26.457879 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:26.458365 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:26.458284 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:26.458992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:26.458976 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:44:28.242510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.242475 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:44:28.309881 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.309059 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-token\" (UniqueName: \"kubernetes.io/projected/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-token\") pod \"8645590d-04a6-4cff-a65e-281d5cb7f059\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " Oct 29 23:44:28.309881 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.309115 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-workload-socket\") pod \"8645590d-04a6-4cff-a65e-281d5cb7f059\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " Oct 29 23:44:28.309881 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.309308 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-podinfo\") pod \"8645590d-04a6-4cff-a65e-281d5cb7f059\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " Oct 29 23:44:28.309881 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.309872 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-workload-certs\") pod \"8645590d-04a6-4cff-a65e-281d5cb7f059\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " Oct 29 23:44:28.314229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.310280 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-workload-certs" pod "8645590d-04a6-4cff-a65e-281d5cb7f059" (UID: "8645590d-04a6-4cff-a65e-281d5cb7f059"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:28.314229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.310647 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-credential-socket" pod "8645590d-04a6-4cff-a65e-281d5cb7f059" (UID: "8645590d-04a6-4cff-a65e-281d5cb7f059"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:28.314229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.311667 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-credential-socket\") pod \"8645590d-04a6-4cff-a65e-281d5cb7f059\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " Oct 29 23:44:28.314229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.311829 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-envoy\") pod \"8645590d-04a6-4cff-a65e-281d5cb7f059\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " Oct 29 23:44:28.314229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.311884 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8645590d-04a6-4cff-a65e-281d5cb7f059-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8645590d-04a6-4cff-a65e-281d5cb7f059-istiod-ca-cert\") pod \"8645590d-04a6-4cff-a65e-281d5cb7f059\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " Oct 29 23:44:28.314229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.312059 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8645590d-04a6-4cff-a65e-281d5cb7f059-kube-api-access-98hfl\" (UniqueName: \"kubernetes.io/projected/8645590d-04a6-4cff-a65e-281d5cb7f059-kube-api-access-98hfl\") pod \"8645590d-04a6-4cff-a65e-281d5cb7f059\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " Oct 29 23:44:28.314229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.312162 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-data\") pod \"8645590d-04a6-4cff-a65e-281d5cb7f059\" (UID: \"8645590d-04a6-4cff-a65e-281d5cb7f059\") " Oct 29 23:44:28.314229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.312267 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8645590d-04a6-4cff-a65e-281d5cb7f059-istiod-ca-cert" pod "8645590d-04a6-4cff-a65e-281d5cb7f059" (UID: "8645590d-04a6-4cff-a65e-281d5cb7f059"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:44:28.314229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.312871 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-data" pod "8645590d-04a6-4cff-a65e-281d5cb7f059" (UID: "8645590d-04a6-4cff-a65e-281d5cb7f059"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:28.314229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.313127 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:28.314229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.313153 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8645590d-04a6-4cff-a65e-281d5cb7f059-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:28.314229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.313169 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:28.314229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.313198 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:28.314953 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.314902 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-workload-socket" pod "8645590d-04a6-4cff-a65e-281d5cb7f059" (UID: "8645590d-04a6-4cff-a65e-281d5cb7f059"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:28.315230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.315105 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-token" pod "8645590d-04a6-4cff-a65e-281d5cb7f059" (UID: "8645590d-04a6-4cff-a65e-281d5cb7f059"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:28.316568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.316543 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-podinfo" pod "8645590d-04a6-4cff-a65e-281d5cb7f059" (UID: "8645590d-04a6-4cff-a65e-281d5cb7f059"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:44:28.316756 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.316739 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8645590d-04a6-4cff-a65e-281d5cb7f059-kube-api-access-98hfl" pod "8645590d-04a6-4cff-a65e-281d5cb7f059" (UID: "8645590d-04a6-4cff-a65e-281d5cb7f059"). InnerVolumeSpecName "kube-api-access-98hfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:28.316945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.316922 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-envoy" pod "8645590d-04a6-4cff-a65e-281d5cb7f059" (UID: "8645590d-04a6-4cff-a65e-281d5cb7f059"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:28.414532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.414453 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98hfl\" (UniqueName: \"kubernetes.io/projected/8645590d-04a6-4cff-a65e-281d5cb7f059-kube-api-access-98hfl\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:28.414532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.414481 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:28.414532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.414491 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:28.414532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.414499 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:28.414532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.414507 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8645590d-04a6-4cff-a65e-281d5cb7f059-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:28.913412 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.913377 5057 generic.go:356] "Generic (PLEG): container finished" podID="8645590d-04a6-4cff-a65e-281d5cb7f059" containerID="3fa14b2805b49b0e8faf99ea4ab05b38ab3b8ec105d0be50397a543aa65a8aa4" exitCode=0 Oct 29 23:44:28.913614 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.913419 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" event={"ID":"8645590d-04a6-4cff-a65e-281d5cb7f059","Type":"ContainerDied","Data":"3fa14b2805b49b0e8faf99ea4ab05b38ab3b8ec105d0be50397a543aa65a8aa4"} Oct 29 23:44:28.913614 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.913485 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" event={"ID":"8645590d-04a6-4cff-a65e-281d5cb7f059","Type":"ContainerDied","Data":"d57ec618f6cdec01c3e0fb8e498c0bf0c9b2fd124439504036e90c11daa4ebc0"} Oct 29 23:44:28.913614 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.913509 5057 scope.go:122] "RemoveContainer" containerID="3fa14b2805b49b0e8faf99ea4ab05b38ab3b8ec105d0be50397a543aa65a8aa4" Oct 29 23:44:28.913614 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.913431 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq" Oct 29 23:44:28.927361 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.927341 5057 scope.go:122] "RemoveContainer" containerID="4c3c6e40a4cae75758e4b48e4f060d61288fe485a7a8bb2d757735edf820d597" Oct 29 23:44:28.939415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.939392 5057 scope.go:122] "RemoveContainer" containerID="3fa14b2805b49b0e8faf99ea4ab05b38ab3b8ec105d0be50397a543aa65a8aa4" Oct 29 23:44:28.939746 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:44:28.939725 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa14b2805b49b0e8faf99ea4ab05b38ab3b8ec105d0be50397a543aa65a8aa4\": container with ID starting with 3fa14b2805b49b0e8faf99ea4ab05b38ab3b8ec105d0be50397a543aa65a8aa4 not found: ID does not exist" containerID="3fa14b2805b49b0e8faf99ea4ab05b38ab3b8ec105d0be50397a543aa65a8aa4" Oct 29 23:44:28.939814 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.939755 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa14b2805b49b0e8faf99ea4ab05b38ab3b8ec105d0be50397a543aa65a8aa4"} err="failed to get container status \"3fa14b2805b49b0e8faf99ea4ab05b38ab3b8ec105d0be50397a543aa65a8aa4\": rpc error: code = NotFound desc = could not find container \"3fa14b2805b49b0e8faf99ea4ab05b38ab3b8ec105d0be50397a543aa65a8aa4\": container with ID starting with 3fa14b2805b49b0e8faf99ea4ab05b38ab3b8ec105d0be50397a543aa65a8aa4 not found: ID does not exist" Oct 29 23:44:28.939814 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.939774 5057 scope.go:122] "RemoveContainer" containerID="4c3c6e40a4cae75758e4b48e4f060d61288fe485a7a8bb2d757735edf820d597" Oct 29 23:44:28.940068 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:44:28.940048 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c3c6e40a4cae75758e4b48e4f060d61288fe485a7a8bb2d757735edf820d597\": container with ID starting with 4c3c6e40a4cae75758e4b48e4f060d61288fe485a7a8bb2d757735edf820d597 not found: ID does not exist" containerID="4c3c6e40a4cae75758e4b48e4f060d61288fe485a7a8bb2d757735edf820d597" Oct 29 23:44:28.940153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.940074 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c3c6e40a4cae75758e4b48e4f060d61288fe485a7a8bb2d757735edf820d597"} err="failed to get container status \"4c3c6e40a4cae75758e4b48e4f060d61288fe485a7a8bb2d757735edf820d597\": rpc error: code = NotFound desc = could not find container \"4c3c6e40a4cae75758e4b48e4f060d61288fe485a7a8bb2d757735edf820d597\": container with ID starting with 4c3c6e40a4cae75758e4b48e4f060d61288fe485a7a8bb2d757735edf820d597 not found: ID does not exist" Oct 29 23:44:28.985899 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.985870 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq"] Oct 29 23:44:28.998045 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:28.998018 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-d8xzq"] Oct 29 23:44:29.037754 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.037719 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt"] Oct 29 23:44:29.038701 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.038678 5057 cpu_manager.go:414] "containerMap: removing container" podUID="8645590d-04a6-4cff-a65e-281d5cb7f059" containerName="istio-proxy" Oct 29 23:44:29.038701 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.038698 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8645590d-04a6-4cff-a65e-281d5cb7f059" containerName="istio-proxy" Oct 29 23:44:29.038892 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.038723 5057 cpu_manager.go:414] "containerMap: removing container" podUID="8645590d-04a6-4cff-a65e-281d5cb7f059" containerName="istio-proxy" Oct 29 23:44:29.038892 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.038728 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8645590d-04a6-4cff-a65e-281d5cb7f059" containerName="istio-proxy" Oct 29 23:44:29.038892 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.038761 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d0f270e1-9bac-4a4c-a472-4d4b07fa7894" containerName="payload-pre-processing" Oct 29 23:44:29.038892 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.038766 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f270e1-9bac-4a4c-a472-4d4b07fa7894" containerName="payload-pre-processing" Oct 29 23:44:29.039101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.038926 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="8645590d-04a6-4cff-a65e-281d5cb7f059" containerName="istio-proxy" Oct 29 23:44:29.039101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.038936 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="8645590d-04a6-4cff-a65e-281d5cb7f059" containerName="istio-proxy" Oct 29 23:44:29.039101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.038950 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f270e1-9bac-4a4c-a472-4d4b07fa7894" containerName="payload-pre-processing" Oct 29 23:44:29.044981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.044948 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" Oct 29 23:44:29.066701 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.066673 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-adc7d7-serving-cert\"" Oct 29 23:44:29.073665 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.073632 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt"] Oct 29 23:44:29.126803 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.126766 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52937560-4419-4c5a-b4a1-1918d3963e22-metrics-certs\") pod \"maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt\" (UID: \"52937560-4419-4c5a-b4a1-1918d3963e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" Oct 29 23:44:29.126989 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.126919 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d64cf\" (UniqueName: \"kubernetes.io/projected/52937560-4419-4c5a-b4a1-1918d3963e22-kube-api-access-d64cf\") pod \"maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt\" (UID: \"52937560-4419-4c5a-b4a1-1918d3963e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" Oct 29 23:44:29.126989 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.126951 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52937560-4419-4c5a-b4a1-1918d3963e22-openshift-service-ca\") pod \"maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt\" (UID: \"52937560-4419-4c5a-b4a1-1918d3963e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" Oct 29 23:44:29.126989 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.126981 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52937560-4419-4c5a-b4a1-1918d3963e22-maas-api-tls\") pod \"maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt\" (UID: \"52937560-4419-4c5a-b4a1-1918d3963e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" Oct 29 23:44:29.228731 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.228685 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52937560-4419-4c5a-b4a1-1918d3963e22-metrics-certs\") pod \"maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt\" (UID: \"52937560-4419-4c5a-b4a1-1918d3963e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" Oct 29 23:44:29.228914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.228830 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d64cf\" (UniqueName: \"kubernetes.io/projected/52937560-4419-4c5a-b4a1-1918d3963e22-kube-api-access-d64cf\") pod \"maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt\" (UID: \"52937560-4419-4c5a-b4a1-1918d3963e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" Oct 29 23:44:29.228914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.228865 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52937560-4419-4c5a-b4a1-1918d3963e22-openshift-service-ca\") pod \"maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt\" (UID: \"52937560-4419-4c5a-b4a1-1918d3963e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" Oct 29 23:44:29.228914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.228897 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52937560-4419-4c5a-b4a1-1918d3963e22-maas-api-tls\") pod \"maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt\" (UID: \"52937560-4419-4c5a-b4a1-1918d3963e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" Oct 29 23:44:29.230095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.230068 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52937560-4419-4c5a-b4a1-1918d3963e22-openshift-service-ca\") pod \"maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt\" (UID: \"52937560-4419-4c5a-b4a1-1918d3963e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" Oct 29 23:44:29.231469 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.231438 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52937560-4419-4c5a-b4a1-1918d3963e22-metrics-certs\") pod \"maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt\" (UID: \"52937560-4419-4c5a-b4a1-1918d3963e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" Oct 29 23:44:29.231952 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.231929 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52937560-4419-4c5a-b4a1-1918d3963e22-maas-api-tls\") pod \"maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt\" (UID: \"52937560-4419-4c5a-b4a1-1918d3963e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" Oct 29 23:44:29.267890 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.262828 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d64cf\" (UniqueName: \"kubernetes.io/projected/52937560-4419-4c5a-b4a1-1918d3963e22-kube-api-access-d64cf\") pod \"maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt\" (UID: \"52937560-4419-4c5a-b4a1-1918d3963e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" Oct 29 23:44:29.371152 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.371122 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" Oct 29 23:44:29.502808 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.502719 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8645590d-04a6-4cff-a65e-281d5cb7f059" path="/var/lib/kubelet/pods/8645590d-04a6-4cff-a65e-281d5cb7f059/volumes" Oct 29 23:44:29.758238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.758180 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt"] Oct 29 23:44:29.759238 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:44:29.759174 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52937560_4419_4c5a_b4a1_1918d3963e22.slice/crio-0831a17c0927e90a1a30e3993f88efb816030e4a35976ed88a8d8f22fc5ce0da WatchSource:0}: Error finding container 0831a17c0927e90a1a30e3993f88efb816030e4a35976ed88a8d8f22fc5ce0da: Status 404 returned error can't find the container with id 0831a17c0927e90a1a30e3993f88efb816030e4a35976ed88a8d8f22fc5ce0da Oct 29 23:44:29.761442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.761424 5057 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:44:29.925437 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:29.925396 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" event={"ID":"52937560-4419-4c5a-b4a1-1918d3963e22","Type":"ContainerStarted","Data":"0831a17c0927e90a1a30e3993f88efb816030e4a35976ed88a8d8f22fc5ce0da"} Oct 29 23:44:30.940823 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:30.940783 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" event={"ID":"52937560-4419-4c5a-b4a1-1918d3963e22","Type":"ContainerStarted","Data":"dda7b678beda825f052de432d5493c23f3dcf75ffb7b72732be39a88096021a9"} Oct 29 23:44:30.942128 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:30.942106 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" Oct 29 23:44:30.986259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:30.986176 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" podStartSLOduration=1.553324486 podStartE2EDuration="1.986161607s" totalImagesPullingTime="432.837121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:44:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:44:30.983889632 +0000 UTC m=+3717.635388162" watchObservedRunningTime="2026-10-29 23:44:30.986161607 +0000 UTC m=+3717.637660137" Oct 29 23:44:33.917041 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:33.917006 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w"] Oct 29 23:44:33.917528 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:44:33.917126 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:44:33.950249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:33.950217 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm"] Oct 29 23:44:38.206659 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:38.206626 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" Oct 29 23:44:38.917358 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:38.917315 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w" podUID="1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3" containerName="payload-processing" containerID="cri-o://2569d769ae96ae35a4402bfb282a353d28c4598e0a4c362ef92c08af5ae54408" gracePeriod=25 Oct 29 23:44:38.951242 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:38.951162 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm" podUID="236cd687-b020-40e9-a69d-7acfcfe2b658" containerName="payload-pre-processing" containerID="cri-o://f2b738c55d8a6c70ad81fffd95885844e5541fc072be53b0a9de799af5693fa3" gracePeriod=25 Oct 29 23:44:39.236631 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:39.236599 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w" Oct 29 23:44:39.329411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:39.329312 5057 generic.go:356] "Generic (PLEG): container finished" podID="1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3" containerID="2569d769ae96ae35a4402bfb282a353d28c4598e0a4c362ef92c08af5ae54408" exitCode=0 Oct 29 23:44:39.329411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:39.329376 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w" Oct 29 23:44:39.329660 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:39.329592 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w" event={"ID":"1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3","Type":"ContainerDied","Data":"2569d769ae96ae35a4402bfb282a353d28c4598e0a4c362ef92c08af5ae54408"} Oct 29 23:44:39.329660 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:39.329639 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w" event={"ID":"1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3","Type":"ContainerDied","Data":"2b278b2bd00f2ffd0f83ff0e1ac12a748eb3394fd602bfc1c4c109ac7be28647"} Oct 29 23:44:39.329787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:39.329670 5057 scope.go:122] "RemoveContainer" containerID="2569d769ae96ae35a4402bfb282a353d28c4598e0a4c362ef92c08af5ae54408" Oct 29 23:44:39.345420 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:39.345393 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3-kube-api-access-xnppc\" (UniqueName: \"kubernetes.io/projected/1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3-kube-api-access-xnppc\") pod \"1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3\" (UID: \"1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3\") " Oct 29 23:44:39.345568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:39.345451 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3-plugins-config-volume\") pod \"1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3\" (UID: \"1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3\") " Oct 29 23:44:39.345923 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:39.345892 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3-plugins-config-volume" pod "1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3" (UID: "1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:44:39.346499 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:39.346473 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:39.347795 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:39.347767 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3-kube-api-access-xnppc" pod "1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3" (UID: "1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3"). InnerVolumeSpecName "kube-api-access-xnppc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:39.348319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:39.348298 5057 scope.go:122] "RemoveContainer" containerID="2569d769ae96ae35a4402bfb282a353d28c4598e0a4c362ef92c08af5ae54408" Oct 29 23:44:39.348638 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:44:39.348618 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2569d769ae96ae35a4402bfb282a353d28c4598e0a4c362ef92c08af5ae54408\": container with ID starting with 2569d769ae96ae35a4402bfb282a353d28c4598e0a4c362ef92c08af5ae54408 not found: ID does not exist" containerID="2569d769ae96ae35a4402bfb282a353d28c4598e0a4c362ef92c08af5ae54408" Oct 29 23:44:39.348782 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:39.348648 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2569d769ae96ae35a4402bfb282a353d28c4598e0a4c362ef92c08af5ae54408"} err="failed to get container status \"2569d769ae96ae35a4402bfb282a353d28c4598e0a4c362ef92c08af5ae54408\": rpc error: code = NotFound desc = could not find container \"2569d769ae96ae35a4402bfb282a353d28c4598e0a4c362ef92c08af5ae54408\": container with ID starting with 2569d769ae96ae35a4402bfb282a353d28c4598e0a4c362ef92c08af5ae54408 not found: ID does not exist" Oct 29 23:44:39.448286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:39.448254 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnppc\" (UniqueName: \"kubernetes.io/projected/1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3-kube-api-access-xnppc\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:39.679447 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:39.679412 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w"] Oct 29 23:44:39.687142 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:39.687103 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6d9e36-5f849b7948-95j4w"] Oct 29 23:44:41.330917 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:41.330882 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7"] Oct 29 23:44:41.331490 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:41.331144 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" podUID="00ddb9f8-8c89-451c-ae16-ae27b0da2485" containerName="istio-proxy" containerID="cri-o://8ba322697e62b2fb98ae9280de624a2418890b7bd6b3206d95d23f3afe7a7ee8" gracePeriod=30 Oct 29 23:44:41.685274 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:41.676910 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3" path="/var/lib/kubelet/pods/1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3/volumes" Oct 29 23:44:43.689083 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:43.689046 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs"] Oct 29 23:44:43.689483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:43.689334 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" podUID="03fd8f1b-0108-4471-a8bd-e1b98dbe8873" containerName="istio-proxy" containerID="cri-o://1aa477f25ae102d5ef50530e01ea8784b1be272256c72f7876df960050258c84" gracePeriod=30 Oct 29 23:44:45.030485 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:45.030442 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm" podUID="236cd687-b020-40e9-a69d-7acfcfe2b658" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:44:46.652380 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.652043 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:44:46.793634 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.793603 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-podinfo\") pod \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " Oct 29 23:44:46.793838 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.793663 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istiod-ca-cert\") pod \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " Oct 29 23:44:46.793838 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.793707 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ddb9f8-8c89-451c-ae16-ae27b0da2485-kube-api-access-f466h\" (UniqueName: \"kubernetes.io/projected/00ddb9f8-8c89-451c-ae16-ae27b0da2485-kube-api-access-f466h\") pod \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " Oct 29 23:44:46.793978 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.793865 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-credential-socket\") pod \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " Oct 29 23:44:46.794043 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.794020 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-workload-certs\") pod \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " Oct 29 23:44:46.794096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.794050 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-data\") pod \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " Oct 29 23:44:46.794157 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.794139 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-envoy\") pod \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " Oct 29 23:44:46.794217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.794179 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istiod-ca-cert" pod "00ddb9f8-8c89-451c-ae16-ae27b0da2485" (UID: "00ddb9f8-8c89-451c-ae16-ae27b0da2485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:44:46.794267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.794232 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-workload-socket\") pod \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " Oct 29 23:44:46.794315 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.794273 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-token\" (UniqueName: \"kubernetes.io/projected/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-token\") pod \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\" (UID: \"00ddb9f8-8c89-451c-ae16-ae27b0da2485\") " Oct 29 23:44:46.794472 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.794440 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-credential-socket" pod "00ddb9f8-8c89-451c-ae16-ae27b0da2485" (UID: "00ddb9f8-8c89-451c-ae16-ae27b0da2485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:46.794918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.794894 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-data" pod "00ddb9f8-8c89-451c-ae16-ae27b0da2485" (UID: "00ddb9f8-8c89-451c-ae16-ae27b0da2485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:46.795179 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.795159 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:46.795847 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.795804 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:46.795847 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.795831 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:46.796025 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.796004 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-workload-socket" pod "00ddb9f8-8c89-451c-ae16-ae27b0da2485" (UID: "00ddb9f8-8c89-451c-ae16-ae27b0da2485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:46.796098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.796081 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-workload-certs" pod "00ddb9f8-8c89-451c-ae16-ae27b0da2485" (UID: "00ddb9f8-8c89-451c-ae16-ae27b0da2485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:46.799773 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.799739 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-podinfo" pod "00ddb9f8-8c89-451c-ae16-ae27b0da2485" (UID: "00ddb9f8-8c89-451c-ae16-ae27b0da2485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:44:46.799888 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.799816 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ddb9f8-8c89-451c-ae16-ae27b0da2485-kube-api-access-f466h" pod "00ddb9f8-8c89-451c-ae16-ae27b0da2485" (UID: "00ddb9f8-8c89-451c-ae16-ae27b0da2485"). InnerVolumeSpecName "kube-api-access-f466h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:46.799968 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.799890 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-token" pod "00ddb9f8-8c89-451c-ae16-ae27b0da2485" (UID: "00ddb9f8-8c89-451c-ae16-ae27b0da2485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:46.801010 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.800989 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-envoy" pod "00ddb9f8-8c89-451c-ae16-ae27b0da2485" (UID: "00ddb9f8-8c89-451c-ae16-ae27b0da2485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:46.897416 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.897332 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:46.897416 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.897361 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:46.897416 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.897370 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ddb9f8-8c89-451c-ae16-ae27b0da2485-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:46.897416 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.897378 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:46.897416 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.897386 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ddb9f8-8c89-451c-ae16-ae27b0da2485-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:46.897416 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:46.897395 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f466h\" (UniqueName: \"kubernetes.io/projected/00ddb9f8-8c89-451c-ae16-ae27b0da2485-kube-api-access-f466h\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:47.411156 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:47.411117 5057 generic.go:356] "Generic (PLEG): container finished" podID="00ddb9f8-8c89-451c-ae16-ae27b0da2485" containerID="8ba322697e62b2fb98ae9280de624a2418890b7bd6b3206d95d23f3afe7a7ee8" exitCode=0 Oct 29 23:44:47.411358 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:47.411168 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" Oct 29 23:44:47.411358 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:47.411230 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" event={"ID":"00ddb9f8-8c89-451c-ae16-ae27b0da2485","Type":"ContainerDied","Data":"8ba322697e62b2fb98ae9280de624a2418890b7bd6b3206d95d23f3afe7a7ee8"} Oct 29 23:44:47.411358 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:47.411288 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7" event={"ID":"00ddb9f8-8c89-451c-ae16-ae27b0da2485","Type":"ContainerDied","Data":"33bbc51a6d67fb24492550aba216c53a7bc2c1ceacce684afa2760d7e0745279"} Oct 29 23:44:47.411358 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:47.411315 5057 scope.go:122] "RemoveContainer" containerID="8ba322697e62b2fb98ae9280de624a2418890b7bd6b3206d95d23f3afe7a7ee8" Oct 29 23:44:47.424744 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:47.424721 5057 scope.go:122] "RemoveContainer" containerID="fe182763aa2258e860aec167ef10a3afd4681e8e494dec7914b71d725d651df2" Oct 29 23:44:47.440347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:47.438443 5057 scope.go:122] "RemoveContainer" containerID="8ba322697e62b2fb98ae9280de624a2418890b7bd6b3206d95d23f3afe7a7ee8" Oct 29 23:44:47.441727 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:44:47.441698 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba322697e62b2fb98ae9280de624a2418890b7bd6b3206d95d23f3afe7a7ee8\": container with ID starting with 8ba322697e62b2fb98ae9280de624a2418890b7bd6b3206d95d23f3afe7a7ee8 not found: ID does not exist" containerID="8ba322697e62b2fb98ae9280de624a2418890b7bd6b3206d95d23f3afe7a7ee8" Oct 29 23:44:47.441874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:47.441739 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba322697e62b2fb98ae9280de624a2418890b7bd6b3206d95d23f3afe7a7ee8"} err="failed to get container status \"8ba322697e62b2fb98ae9280de624a2418890b7bd6b3206d95d23f3afe7a7ee8\": rpc error: code = NotFound desc = could not find container \"8ba322697e62b2fb98ae9280de624a2418890b7bd6b3206d95d23f3afe7a7ee8\": container with ID starting with 8ba322697e62b2fb98ae9280de624a2418890b7bd6b3206d95d23f3afe7a7ee8 not found: ID does not exist" Oct 29 23:44:47.441874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:47.441762 5057 scope.go:122] "RemoveContainer" containerID="fe182763aa2258e860aec167ef10a3afd4681e8e494dec7914b71d725d651df2" Oct 29 23:44:47.442212 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:44:47.442169 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe182763aa2258e860aec167ef10a3afd4681e8e494dec7914b71d725d651df2\": container with ID starting with fe182763aa2258e860aec167ef10a3afd4681e8e494dec7914b71d725d651df2 not found: ID does not exist" containerID="fe182763aa2258e860aec167ef10a3afd4681e8e494dec7914b71d725d651df2" Oct 29 23:44:47.442344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:47.442222 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe182763aa2258e860aec167ef10a3afd4681e8e494dec7914b71d725d651df2"} err="failed to get container status \"fe182763aa2258e860aec167ef10a3afd4681e8e494dec7914b71d725d651df2\": rpc error: code = NotFound desc = could not find container \"fe182763aa2258e860aec167ef10a3afd4681e8e494dec7914b71d725d651df2\": container with ID starting with fe182763aa2258e860aec167ef10a3afd4681e8e494dec7914b71d725d651df2 not found: ID does not exist" Oct 29 23:44:47.468892 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:47.468870 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7"] Oct 29 23:44:47.486331 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:47.486305 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d5e482-openshift-default-6cbfdb5dfc-nxtk7"] Oct 29 23:44:48.543357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:48.543321 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4"] Oct 29 23:44:49.054601 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.054566 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:44:49.282214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.281722 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-workload-certs\") pod \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " Oct 29 23:44:49.282214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.281796 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istiod-ca-cert\") pod \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " Oct 29 23:44:49.282214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.281840 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-workload-socket\") pod \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " Oct 29 23:44:49.282214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.281982 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-envoy\") pod \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " Oct 29 23:44:49.282214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.282083 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-token\" (UniqueName: \"kubernetes.io/projected/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-token\") pod \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " Oct 29 23:44:49.282214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.282134 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-kube-api-access-b2fbg\" (UniqueName: \"kubernetes.io/projected/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-kube-api-access-b2fbg\") pod \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " Oct 29 23:44:49.284024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.282828 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-credential-socket\") pod \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " Oct 29 23:44:49.284024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.282971 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-podinfo\") pod \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " Oct 29 23:44:49.284024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.283008 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-data\") pod \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\" (UID: \"03fd8f1b-0108-4471-a8bd-e1b98dbe8873\") " Oct 29 23:44:49.284504 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.284309 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-data" pod "03fd8f1b-0108-4471-a8bd-e1b98dbe8873" (UID: "03fd8f1b-0108-4471-a8bd-e1b98dbe8873"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:49.284847 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.284629 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-workload-certs" pod "03fd8f1b-0108-4471-a8bd-e1b98dbe8873" (UID: "03fd8f1b-0108-4471-a8bd-e1b98dbe8873"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:49.286210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.285071 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istiod-ca-cert" pod "03fd8f1b-0108-4471-a8bd-e1b98dbe8873" (UID: "03fd8f1b-0108-4471-a8bd-e1b98dbe8873"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:44:49.286695 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.286364 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-workload-socket" pod "03fd8f1b-0108-4471-a8bd-e1b98dbe8873" (UID: "03fd8f1b-0108-4471-a8bd-e1b98dbe8873"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:49.291897 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.291635 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-credential-socket" pod "03fd8f1b-0108-4471-a8bd-e1b98dbe8873" (UID: "03fd8f1b-0108-4471-a8bd-e1b98dbe8873"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:49.301397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.301374 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-envoy" pod "03fd8f1b-0108-4471-a8bd-e1b98dbe8873" (UID: "03fd8f1b-0108-4471-a8bd-e1b98dbe8873"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:49.332959 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.332900 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-token" pod "03fd8f1b-0108-4471-a8bd-e1b98dbe8873" (UID: "03fd8f1b-0108-4471-a8bd-e1b98dbe8873"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:49.333635 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.333592 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-podinfo" pod "03fd8f1b-0108-4471-a8bd-e1b98dbe8873" (UID: "03fd8f1b-0108-4471-a8bd-e1b98dbe8873"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:44:49.344967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.344928 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-kube-api-access-b2fbg" pod "03fd8f1b-0108-4471-a8bd-e1b98dbe8873" (UID: "03fd8f1b-0108-4471-a8bd-e1b98dbe8873"). InnerVolumeSpecName "kube-api-access-b2fbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:49.421210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.414500 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:49.421210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.414536 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:49.421210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.414550 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:49.421210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.414564 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:49.421210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.414577 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:49.421210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.414597 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:49.421210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.414611 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:49.421210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.414624 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:49.421210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.414636 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2fbg\" (UniqueName: \"kubernetes.io/projected/03fd8f1b-0108-4471-a8bd-e1b98dbe8873-kube-api-access-b2fbg\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:49.510209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.510154 5057 generic.go:356] "Generic (PLEG): container finished" podID="03fd8f1b-0108-4471-a8bd-e1b98dbe8873" containerID="1aa477f25ae102d5ef50530e01ea8784b1be272256c72f7876df960050258c84" exitCode=0 Oct 29 23:44:49.510666 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.510558 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" Oct 29 23:44:49.513955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.513922 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00ddb9f8-8c89-451c-ae16-ae27b0da2485" path="/var/lib/kubelet/pods/00ddb9f8-8c89-451c-ae16-ae27b0da2485/volumes" Oct 29 23:44:49.514848 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.514814 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" event={"ID":"03fd8f1b-0108-4471-a8bd-e1b98dbe8873","Type":"ContainerDied","Data":"1aa477f25ae102d5ef50530e01ea8784b1be272256c72f7876df960050258c84"} Oct 29 23:44:49.514980 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.514875 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs" event={"ID":"03fd8f1b-0108-4471-a8bd-e1b98dbe8873","Type":"ContainerDied","Data":"d564e69d6dd391e8ffbfe29a6c5ce4106375254d008d2e39c50195bed9423851"} Oct 29 23:44:49.514980 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.514907 5057 scope.go:122] "RemoveContainer" containerID="1aa477f25ae102d5ef50530e01ea8784b1be272256c72f7876df960050258c84" Oct 29 23:44:49.538330 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.538171 5057 scope.go:122] "RemoveContainer" containerID="0106c8f7f7cfe71fdfb41aa0e9bc475858acd8cde0c57ecc59c810a0ca037e2f" Oct 29 23:44:49.551267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.551248 5057 scope.go:122] "RemoveContainer" containerID="1aa477f25ae102d5ef50530e01ea8784b1be272256c72f7876df960050258c84" Oct 29 23:44:49.551646 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:44:49.551580 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa477f25ae102d5ef50530e01ea8784b1be272256c72f7876df960050258c84\": container with ID starting with 1aa477f25ae102d5ef50530e01ea8784b1be272256c72f7876df960050258c84 not found: ID does not exist" containerID="1aa477f25ae102d5ef50530e01ea8784b1be272256c72f7876df960050258c84" Oct 29 23:44:49.551646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.551604 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa477f25ae102d5ef50530e01ea8784b1be272256c72f7876df960050258c84"} err="failed to get container status \"1aa477f25ae102d5ef50530e01ea8784b1be272256c72f7876df960050258c84\": rpc error: code = NotFound desc = could not find container \"1aa477f25ae102d5ef50530e01ea8784b1be272256c72f7876df960050258c84\": container with ID starting with 1aa477f25ae102d5ef50530e01ea8784b1be272256c72f7876df960050258c84 not found: ID does not exist" Oct 29 23:44:49.551646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.551621 5057 scope.go:122] "RemoveContainer" containerID="0106c8f7f7cfe71fdfb41aa0e9bc475858acd8cde0c57ecc59c810a0ca037e2f" Oct 29 23:44:49.551864 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:44:49.551843 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0106c8f7f7cfe71fdfb41aa0e9bc475858acd8cde0c57ecc59c810a0ca037e2f\": container with ID starting with 0106c8f7f7cfe71fdfb41aa0e9bc475858acd8cde0c57ecc59c810a0ca037e2f not found: ID does not exist" containerID="0106c8f7f7cfe71fdfb41aa0e9bc475858acd8cde0c57ecc59c810a0ca037e2f" Oct 29 23:44:49.551936 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.551867 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0106c8f7f7cfe71fdfb41aa0e9bc475858acd8cde0c57ecc59c810a0ca037e2f"} err="failed to get container status \"0106c8f7f7cfe71fdfb41aa0e9bc475858acd8cde0c57ecc59c810a0ca037e2f\": rpc error: code = NotFound desc = could not find container \"0106c8f7f7cfe71fdfb41aa0e9bc475858acd8cde0c57ecc59c810a0ca037e2f\": container with ID starting with 0106c8f7f7cfe71fdfb41aa0e9bc475858acd8cde0c57ecc59c810a0ca037e2f not found: ID does not exist" Oct 29 23:44:49.572027 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.571990 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs"] Oct 29 23:44:49.581369 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:49.581340 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6d9e36-openshift-default-75947cfcf9-ggxrs"] Oct 29 23:44:51.484554 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:51.484523 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03fd8f1b-0108-4471-a8bd-e1b98dbe8873" path="/var/lib/kubelet/pods/03fd8f1b-0108-4471-a8bd-e1b98dbe8873/volumes" Oct 29 23:44:53.543911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:53.543862 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4" podUID="b7efd7af-6379-4f18-b4e0-6ed403891e10" containerName="payload-pre-processing" containerID="cri-o://cc4ce5889b8787157cbbf4ea5afdabd8192cd2e9240ab8e979d35561d7b21442" gracePeriod=25 Oct 29 23:44:54.233733 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:54.233704 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4" Oct 29 23:44:54.285559 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:54.285525 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7efd7af-6379-4f18-b4e0-6ed403891e10-kube-api-access-pltm6\" (UniqueName: \"kubernetes.io/projected/b7efd7af-6379-4f18-b4e0-6ed403891e10-kube-api-access-pltm6\") pod \"b7efd7af-6379-4f18-b4e0-6ed403891e10\" (UID: \"b7efd7af-6379-4f18-b4e0-6ed403891e10\") " Oct 29 23:44:54.285760 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:54.285597 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7efd7af-6379-4f18-b4e0-6ed403891e10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7efd7af-6379-4f18-b4e0-6ed403891e10-plugins-config-volume\") pod \"b7efd7af-6379-4f18-b4e0-6ed403891e10\" (UID: \"b7efd7af-6379-4f18-b4e0-6ed403891e10\") " Oct 29 23:44:54.286053 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:54.286027 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7efd7af-6379-4f18-b4e0-6ed403891e10-plugins-config-volume" pod "b7efd7af-6379-4f18-b4e0-6ed403891e10" (UID: "b7efd7af-6379-4f18-b4e0-6ed403891e10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:44:54.286349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:54.286330 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7efd7af-6379-4f18-b4e0-6ed403891e10-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:54.287920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:54.287893 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7efd7af-6379-4f18-b4e0-6ed403891e10-kube-api-access-pltm6" pod "b7efd7af-6379-4f18-b4e0-6ed403891e10" (UID: "b7efd7af-6379-4f18-b4e0-6ed403891e10"). InnerVolumeSpecName "kube-api-access-pltm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:54.388343 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:54.388297 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pltm6\" (UniqueName: \"kubernetes.io/projected/b7efd7af-6379-4f18-b4e0-6ed403891e10-kube-api-access-pltm6\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:54.600805 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:54.600720 5057 generic.go:356] "Generic (PLEG): container finished" podID="b7efd7af-6379-4f18-b4e0-6ed403891e10" containerID="cc4ce5889b8787157cbbf4ea5afdabd8192cd2e9240ab8e979d35561d7b21442" exitCode=0 Oct 29 23:44:54.600805 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:54.600773 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4" Oct 29 23:44:54.601307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:54.600800 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4" event={"ID":"b7efd7af-6379-4f18-b4e0-6ed403891e10","Type":"ContainerDied","Data":"cc4ce5889b8787157cbbf4ea5afdabd8192cd2e9240ab8e979d35561d7b21442"} Oct 29 23:44:54.601307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:54.600856 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4" event={"ID":"b7efd7af-6379-4f18-b4e0-6ed403891e10","Type":"ContainerDied","Data":"1719f878a79f6fe10dd2e07db72549bcdefe10c1b417b6c2a15ef89b5e2c230e"} Oct 29 23:44:54.601307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:54.600883 5057 scope.go:122] "RemoveContainer" containerID="cc4ce5889b8787157cbbf4ea5afdabd8192cd2e9240ab8e979d35561d7b21442" Oct 29 23:44:54.613463 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:54.613440 5057 scope.go:122] "RemoveContainer" containerID="cc4ce5889b8787157cbbf4ea5afdabd8192cd2e9240ab8e979d35561d7b21442" Oct 29 23:44:54.613815 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:44:54.613791 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc4ce5889b8787157cbbf4ea5afdabd8192cd2e9240ab8e979d35561d7b21442\": container with ID starting with cc4ce5889b8787157cbbf4ea5afdabd8192cd2e9240ab8e979d35561d7b21442 not found: ID does not exist" containerID="cc4ce5889b8787157cbbf4ea5afdabd8192cd2e9240ab8e979d35561d7b21442" Oct 29 23:44:54.613895 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:54.613828 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc4ce5889b8787157cbbf4ea5afdabd8192cd2e9240ab8e979d35561d7b21442"} err="failed to get container status \"cc4ce5889b8787157cbbf4ea5afdabd8192cd2e9240ab8e979d35561d7b21442\": rpc error: code = NotFound desc = could not find container \"cc4ce5889b8787157cbbf4ea5afdabd8192cd2e9240ab8e979d35561d7b21442\": container with ID starting with cc4ce5889b8787157cbbf4ea5afdabd8192cd2e9240ab8e979d35561d7b21442 not found: ID does not exist" Oct 29 23:44:54.663216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:54.660297 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4"] Oct 29 23:44:54.671309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:54.671265 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-55c3fc-67d5ccd66c-bnxf4"] Oct 29 23:44:55.030502 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:55.030456 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm" podUID="236cd687-b020-40e9-a69d-7acfcfe2b658" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:44:55.522468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:55.522429 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7efd7af-6379-4f18-b4e0-6ed403891e10" path="/var/lib/kubelet/pods/b7efd7af-6379-4f18-b4e0-6ed403891e10/volumes" Oct 29 23:44:59.269793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:59.269763 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm" Oct 29 23:44:59.300172 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:59.300144 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/236cd687-b020-40e9-a69d-7acfcfe2b658-kube-api-access-l5xjq\" (UniqueName: \"kubernetes.io/projected/236cd687-b020-40e9-a69d-7acfcfe2b658-kube-api-access-l5xjq\") pod \"236cd687-b020-40e9-a69d-7acfcfe2b658\" (UID: \"236cd687-b020-40e9-a69d-7acfcfe2b658\") " Oct 29 23:44:59.300439 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:59.300422 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/236cd687-b020-40e9-a69d-7acfcfe2b658-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/236cd687-b020-40e9-a69d-7acfcfe2b658-plugins-config-volume\") pod \"236cd687-b020-40e9-a69d-7acfcfe2b658\" (UID: \"236cd687-b020-40e9-a69d-7acfcfe2b658\") " Oct 29 23:44:59.300762 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:59.300735 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/236cd687-b020-40e9-a69d-7acfcfe2b658-plugins-config-volume" pod "236cd687-b020-40e9-a69d-7acfcfe2b658" (UID: "236cd687-b020-40e9-a69d-7acfcfe2b658"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:44:59.301142 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:59.301121 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/236cd687-b020-40e9-a69d-7acfcfe2b658-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:59.302361 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:59.302336 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/236cd687-b020-40e9-a69d-7acfcfe2b658-kube-api-access-l5xjq" pod "236cd687-b020-40e9-a69d-7acfcfe2b658" (UID: "236cd687-b020-40e9-a69d-7acfcfe2b658"). InnerVolumeSpecName "kube-api-access-l5xjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:59.402786 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:59.402754 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5xjq\" (UniqueName: \"kubernetes.io/projected/236cd687-b020-40e9-a69d-7acfcfe2b658-kube-api-access-l5xjq\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:44:59.680139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:59.680106 5057 generic.go:356] "Generic (PLEG): container finished" podID="236cd687-b020-40e9-a69d-7acfcfe2b658" containerID="f2b738c55d8a6c70ad81fffd95885844e5541fc072be53b0a9de799af5693fa3" exitCode=0 Oct 29 23:44:59.680349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:59.680146 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm" Oct 29 23:44:59.680349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:59.680225 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm" event={"ID":"236cd687-b020-40e9-a69d-7acfcfe2b658","Type":"ContainerDied","Data":"f2b738c55d8a6c70ad81fffd95885844e5541fc072be53b0a9de799af5693fa3"} Oct 29 23:44:59.680349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:59.680286 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm" event={"ID":"236cd687-b020-40e9-a69d-7acfcfe2b658","Type":"ContainerDied","Data":"132cf1e863e7013d3ca0e6b7c2b7c5ffd16d901ea530c607db50883ee28f7328"} Oct 29 23:44:59.680349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:59.680309 5057 scope.go:122] "RemoveContainer" containerID="f2b738c55d8a6c70ad81fffd95885844e5541fc072be53b0a9de799af5693fa3" Oct 29 23:44:59.695005 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:59.694985 5057 scope.go:122] "RemoveContainer" containerID="f2b738c55d8a6c70ad81fffd95885844e5541fc072be53b0a9de799af5693fa3" Oct 29 23:44:59.695302 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:44:59.695280 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2b738c55d8a6c70ad81fffd95885844e5541fc072be53b0a9de799af5693fa3\": container with ID starting with f2b738c55d8a6c70ad81fffd95885844e5541fc072be53b0a9de799af5693fa3 not found: ID does not exist" containerID="f2b738c55d8a6c70ad81fffd95885844e5541fc072be53b0a9de799af5693fa3" Oct 29 23:44:59.695402 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:59.695313 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b738c55d8a6c70ad81fffd95885844e5541fc072be53b0a9de799af5693fa3"} err="failed to get container status \"f2b738c55d8a6c70ad81fffd95885844e5541fc072be53b0a9de799af5693fa3\": rpc error: code = NotFound desc = could not find container \"f2b738c55d8a6c70ad81fffd95885844e5541fc072be53b0a9de799af5693fa3\": container with ID starting with f2b738c55d8a6c70ad81fffd95885844e5541fc072be53b0a9de799af5693fa3 not found: ID does not exist" Oct 29 23:44:59.736202 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:59.736164 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm"] Oct 29 23:44:59.748008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:44:59.747925 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6d9e36-67dbdff55f-b72zm"] Oct 29 23:45:00.154815 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.154720 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc"] Oct 29 23:45:00.156105 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.156077 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3" containerName="payload-processing" Oct 29 23:45:00.156105 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.156103 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3" containerName="payload-processing" Oct 29 23:45:00.156319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.156123 5057 cpu_manager.go:414] "containerMap: removing container" podUID="03fd8f1b-0108-4471-a8bd-e1b98dbe8873" containerName="istio-proxy" Oct 29 23:45:00.156319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.156130 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03fd8f1b-0108-4471-a8bd-e1b98dbe8873" containerName="istio-proxy" Oct 29 23:45:00.156319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.156142 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b7efd7af-6379-4f18-b4e0-6ed403891e10" containerName="payload-pre-processing" Oct 29 23:45:00.156319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.156152 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7efd7af-6379-4f18-b4e0-6ed403891e10" containerName="payload-pre-processing" Oct 29 23:45:00.156319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.156203 5057 cpu_manager.go:414] "containerMap: removing container" podUID="236cd687-b020-40e9-a69d-7acfcfe2b658" containerName="payload-pre-processing" Oct 29 23:45:00.156319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.156211 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="236cd687-b020-40e9-a69d-7acfcfe2b658" containerName="payload-pre-processing" Oct 29 23:45:00.156319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.156236 5057 cpu_manager.go:414] "containerMap: removing container" podUID="03fd8f1b-0108-4471-a8bd-e1b98dbe8873" containerName="istio-proxy" Oct 29 23:45:00.156319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.156243 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03fd8f1b-0108-4471-a8bd-e1b98dbe8873" containerName="istio-proxy" Oct 29 23:45:00.156319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.156278 5057 cpu_manager.go:414] "containerMap: removing container" podUID="00ddb9f8-8c89-451c-ae16-ae27b0da2485" containerName="istio-proxy" Oct 29 23:45:00.156319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.156285 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ddb9f8-8c89-451c-ae16-ae27b0da2485" containerName="istio-proxy" Oct 29 23:45:00.156806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.156327 5057 cpu_manager.go:414] "containerMap: removing container" podUID="00ddb9f8-8c89-451c-ae16-ae27b0da2485" containerName="istio-proxy" Oct 29 23:45:00.156806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.156334 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ddb9f8-8c89-451c-ae16-ae27b0da2485" containerName="istio-proxy" Oct 29 23:45:00.156806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.156551 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="236cd687-b020-40e9-a69d-7acfcfe2b658" containerName="payload-pre-processing" Oct 29 23:45:00.156806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.156567 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="03fd8f1b-0108-4471-a8bd-e1b98dbe8873" containerName="istio-proxy" Oct 29 23:45:00.156806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.156583 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="00ddb9f8-8c89-451c-ae16-ae27b0da2485" containerName="istio-proxy" Oct 29 23:45:00.156806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.156592 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="00ddb9f8-8c89-451c-ae16-ae27b0da2485" containerName="istio-proxy" Oct 29 23:45:00.156806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.156602 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f6d1edb-6a64-45e0-96f4-b77ea25ebbf3" containerName="payload-processing" Oct 29 23:45:00.156806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.156608 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7efd7af-6379-4f18-b4e0-6ed403891e10" containerName="payload-pre-processing" Oct 29 23:45:00.162841 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.162817 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc" Oct 29 23:45:00.189740 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.189704 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc"] Oct 29 23:45:00.195062 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.195032 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z"] Oct 29 23:45:00.196978 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.196950 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="03fd8f1b-0108-4471-a8bd-e1b98dbe8873" containerName="istio-proxy" Oct 29 23:45:00.208255 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.202496 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5"] Oct 29 23:45:00.208255 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.202634 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" Oct 29 23:45:00.210630 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.210601 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" Oct 29 23:45:00.214952 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.214912 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5"] Oct 29 23:45:00.226487 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.226458 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" Oct 29 23:45:00.227703 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.227671 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z"] Oct 29 23:45:00.231980 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.231949 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b0ed631-9976-4f08-8d68-e783428c6a48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc\" (UID: \"2b0ed631-9976-4f08-8d68-e783428c6a48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc" Oct 29 23:45:00.232171 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.232153 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmh49\" (UniqueName: \"kubernetes.io/projected/2b0ed631-9976-4f08-8d68-e783428c6a48-kube-api-access-tmh49\") pod \"maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc\" (UID: \"2b0ed631-9976-4f08-8d68-e783428c6a48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc" Oct 29 23:45:00.242554 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.242518 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5"] Oct 29 23:45:00.255052 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.255025 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6"] Oct 29 23:45:00.272558 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.271724 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:45:00.283257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.283082 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz"] Oct 29 23:45:00.286035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.286009 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6" Oct 29 23:45:00.308577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.308546 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5"] Oct 29 23:45:00.308738 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.308620 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz"] Oct 29 23:45:00.308738 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.308658 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh"] Oct 29 23:45:00.309258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.309234 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" Oct 29 23:45:00.335660 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.335621 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/759dfe21-e08a-42ae-a63d-097ada97aa15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5\" (UID: \"759dfe21-e08a-42ae-a63d-097ada97aa15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" Oct 29 23:45:00.335837 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.335787 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b0ed631-9976-4f08-8d68-e783428c6a48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc\" (UID: \"2b0ed631-9976-4f08-8d68-e783428c6a48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc" Oct 29 23:45:00.335837 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.335824 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmh49\" (UniqueName: \"kubernetes.io/projected/2b0ed631-9976-4f08-8d68-e783428c6a48-kube-api-access-tmh49\") pod \"maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc\" (UID: \"2b0ed631-9976-4f08-8d68-e783428c6a48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc" Oct 29 23:45:00.335968 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.335882 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fce6be-41f6-4874-b280-6f0422b87a06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z\" (UID: \"01fce6be-41f6-4874-b280-6f0422b87a06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" Oct 29 23:45:00.335968 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.335940 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpdgw\" (UniqueName: \"kubernetes.io/projected/848ba138-a799-4dab-b704-2ae74e04ac95-kube-api-access-vpdgw\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5\" (UID: \"848ba138-a799-4dab-b704-2ae74e04ac95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" Oct 29 23:45:00.336087 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.335985 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlcw7\" (UniqueName: \"kubernetes.io/projected/8496326e-aae6-49fc-be66-4cdeb78ccf74-kube-api-access-zlcw7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6\" (UID: \"8496326e-aae6-49fc-be66-4cdeb78ccf74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6" Oct 29 23:45:00.336161 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.336146 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdj94\" (UniqueName: \"kubernetes.io/projected/01fce6be-41f6-4874-b280-6f0422b87a06-kube-api-access-kdj94\") pod \"maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z\" (UID: \"01fce6be-41f6-4874-b280-6f0422b87a06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" Oct 29 23:45:00.336325 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.336308 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj9gh\" (UniqueName: \"kubernetes.io/projected/759dfe21-e08a-42ae-a63d-097ada97aa15-kube-api-access-zj9gh\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5\" (UID: \"759dfe21-e08a-42ae-a63d-097ada97aa15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" Oct 29 23:45:00.336413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.336395 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8496326e-aae6-49fc-be66-4cdeb78ccf74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6\" (UID: \"8496326e-aae6-49fc-be66-4cdeb78ccf74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6" Oct 29 23:45:00.336503 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.336490 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/848ba138-a799-4dab-b704-2ae74e04ac95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5\" (UID: \"848ba138-a799-4dab-b704-2ae74e04ac95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" Oct 29 23:45:00.339252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.337486 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b0ed631-9976-4f08-8d68-e783428c6a48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc\" (UID: \"2b0ed631-9976-4f08-8d68-e783428c6a48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc" Oct 29 23:45:00.350255 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.343587 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" Oct 29 23:45:00.350255 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.345217 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5"] Oct 29 23:45:00.354929 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.354867 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" Oct 29 23:45:00.385671 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.384789 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmh49\" (UniqueName: \"kubernetes.io/projected/2b0ed631-9976-4f08-8d68-e783428c6a48-kube-api-access-tmh49\") pod \"maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc\" (UID: \"2b0ed631-9976-4f08-8d68-e783428c6a48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc" Oct 29 23:45:00.385671 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.385197 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh"] Oct 29 23:45:00.414300 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.413068 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5"] Oct 29 23:45:00.427717 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.427395 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb"] Oct 29 23:45:00.457909 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.452164 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdj94\" (UniqueName: \"kubernetes.io/projected/01fce6be-41f6-4874-b280-6f0422b87a06-kube-api-access-kdj94\") pod \"maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z\" (UID: \"01fce6be-41f6-4874-b280-6f0422b87a06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" Oct 29 23:45:00.457909 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.452248 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2hhr\" (UniqueName: \"kubernetes.io/projected/1d322efb-9ed7-4868-b1b5-6ac26e0ce34d-kube-api-access-j2hhr\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh\" (UID: \"1d322efb-9ed7-4868-b1b5-6ac26e0ce34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" Oct 29 23:45:00.461605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.461540 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5"] Oct 29 23:45:00.462033 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.461870 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb" Oct 29 23:45:00.473701 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.452292 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da38ccbd-b489-4225-a4d1-4502943eb3d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz\" (UID: \"da38ccbd-b489-4225-a4d1-4502943eb3d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" Oct 29 23:45:00.473843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.473815 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj9gh\" (UniqueName: \"kubernetes.io/projected/759dfe21-e08a-42ae-a63d-097ada97aa15-kube-api-access-zj9gh\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5\" (UID: \"759dfe21-e08a-42ae-a63d-097ada97aa15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" Oct 29 23:45:00.473910 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.473901 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8496326e-aae6-49fc-be66-4cdeb78ccf74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6\" (UID: \"8496326e-aae6-49fc-be66-4cdeb78ccf74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6" Oct 29 23:45:00.474011 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.473995 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zmgd\" (UniqueName: \"kubernetes.io/projected/e2707a1f-9602-475e-83ea-df56bcf01224-kube-api-access-5zmgd\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5\" (UID: \"e2707a1f-9602-475e-83ea-df56bcf01224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" Oct 29 23:45:00.474090 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.474032 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/848ba138-a799-4dab-b704-2ae74e04ac95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5\" (UID: \"848ba138-a799-4dab-b704-2ae74e04ac95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" Oct 29 23:45:00.474153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.474087 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/759dfe21-e08a-42ae-a63d-097ada97aa15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5\" (UID: \"759dfe21-e08a-42ae-a63d-097ada97aa15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" Oct 29 23:45:00.474153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.474140 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2707a1f-9602-475e-83ea-df56bcf01224-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5\" (UID: \"e2707a1f-9602-475e-83ea-df56bcf01224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" Oct 29 23:45:00.474290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.474164 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkkkc\" (UniqueName: \"kubernetes.io/projected/da38ccbd-b489-4225-a4d1-4502943eb3d4-kube-api-access-kkkkc\") pod \"maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz\" (UID: \"da38ccbd-b489-4225-a4d1-4502943eb3d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" Oct 29 23:45:00.474605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.474555 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d322efb-9ed7-4868-b1b5-6ac26e0ce34d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh\" (UID: \"1d322efb-9ed7-4868-b1b5-6ac26e0ce34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" Oct 29 23:45:00.474732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.474714 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fce6be-41f6-4874-b280-6f0422b87a06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z\" (UID: \"01fce6be-41f6-4874-b280-6f0422b87a06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" Oct 29 23:45:00.474804 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.474783 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpdgw\" (UniqueName: \"kubernetes.io/projected/848ba138-a799-4dab-b704-2ae74e04ac95-kube-api-access-vpdgw\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5\" (UID: \"848ba138-a799-4dab-b704-2ae74e04ac95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" Oct 29 23:45:00.474866 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.474833 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlcw7\" (UniqueName: \"kubernetes.io/projected/8496326e-aae6-49fc-be66-4cdeb78ccf74-kube-api-access-zlcw7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6\" (UID: \"8496326e-aae6-49fc-be66-4cdeb78ccf74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6" Oct 29 23:45:00.476161 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.476134 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8496326e-aae6-49fc-be66-4cdeb78ccf74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6\" (UID: \"8496326e-aae6-49fc-be66-4cdeb78ccf74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6" Oct 29 23:45:00.476946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.476885 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fce6be-41f6-4874-b280-6f0422b87a06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z\" (UID: \"01fce6be-41f6-4874-b280-6f0422b87a06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" Oct 29 23:45:00.478906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.476961 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/848ba138-a799-4dab-b704-2ae74e04ac95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5\" (UID: \"848ba138-a799-4dab-b704-2ae74e04ac95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" Oct 29 23:45:00.478906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.477740 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/759dfe21-e08a-42ae-a63d-097ada97aa15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5\" (UID: \"759dfe21-e08a-42ae-a63d-097ada97aa15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" Oct 29 23:45:00.480786 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.480770 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc" Oct 29 23:45:00.489939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.486174 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6"] Oct 29 23:45:00.489939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.486286 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5"] Oct 29 23:45:00.489939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.486312 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb"] Oct 29 23:45:00.489939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.486391 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" Oct 29 23:45:00.489939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.489412 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdj94\" (UniqueName: \"kubernetes.io/projected/01fce6be-41f6-4874-b280-6f0422b87a06-kube-api-access-kdj94\") pod \"maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z\" (UID: \"01fce6be-41f6-4874-b280-6f0422b87a06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" Oct 29 23:45:00.499821 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.499797 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b"] Oct 29 23:45:00.511003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.508078 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj9gh\" (UniqueName: \"kubernetes.io/projected/759dfe21-e08a-42ae-a63d-097ada97aa15-kube-api-access-zj9gh\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5\" (UID: \"759dfe21-e08a-42ae-a63d-097ada97aa15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" Oct 29 23:45:00.512367 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.512340 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r"] Oct 29 23:45:00.513549 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.513530 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" Oct 29 23:45:00.534865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.527135 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj"] Oct 29 23:45:00.534865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.532004 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" Oct 29 23:45:00.534865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.533487 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpdgw\" (UniqueName: \"kubernetes.io/projected/848ba138-a799-4dab-b704-2ae74e04ac95-kube-api-access-vpdgw\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5\" (UID: \"848ba138-a799-4dab-b704-2ae74e04ac95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" Oct 29 23:45:00.534865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.534433 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlcw7\" (UniqueName: \"kubernetes.io/projected/8496326e-aae6-49fc-be66-4cdeb78ccf74-kube-api-access-zlcw7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6\" (UID: \"8496326e-aae6-49fc-be66-4cdeb78ccf74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6" Oct 29 23:45:00.538089 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.537974 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" Oct 29 23:45:00.538615 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.538595 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj" Oct 29 23:45:00.540533 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.540507 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b"] Oct 29 23:45:00.548223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.547644 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv"] Oct 29 23:45:00.557702 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.557682 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r"] Oct 29 23:45:00.557950 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.557935 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" Oct 29 23:45:00.570338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.570251 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj"] Oct 29 23:45:00.576605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.576364 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58tkn\" (UniqueName: \"kubernetes.io/projected/2c26d97b-29bd-4a99-813a-91cd764b3199-kube-api-access-58tkn\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5\" (UID: \"2c26d97b-29bd-4a99-813a-91cd764b3199\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" Oct 29 23:45:00.576605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.576426 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2hhr\" (UniqueName: \"kubernetes.io/projected/1d322efb-9ed7-4868-b1b5-6ac26e0ce34d-kube-api-access-j2hhr\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh\" (UID: \"1d322efb-9ed7-4868-b1b5-6ac26e0ce34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" Oct 29 23:45:00.576605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.576457 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da38ccbd-b489-4225-a4d1-4502943eb3d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz\" (UID: \"da38ccbd-b489-4225-a4d1-4502943eb3d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" Oct 29 23:45:00.576605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.576537 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c26d97b-29bd-4a99-813a-91cd764b3199-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5\" (UID: \"2c26d97b-29bd-4a99-813a-91cd764b3199\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" Oct 29 23:45:00.576605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.576555 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkjqc\" (UniqueName: \"kubernetes.io/projected/f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142-kube-api-access-dkjqc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb\" (UID: \"f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb" Oct 29 23:45:00.579213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.576907 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zmgd\" (UniqueName: \"kubernetes.io/projected/e2707a1f-9602-475e-83ea-df56bcf01224-kube-api-access-5zmgd\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5\" (UID: \"e2707a1f-9602-475e-83ea-df56bcf01224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" Oct 29 23:45:00.579213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.576972 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb\" (UID: \"f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb" Oct 29 23:45:00.579213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.577001 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2707a1f-9602-475e-83ea-df56bcf01224-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5\" (UID: \"e2707a1f-9602-475e-83ea-df56bcf01224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" Oct 29 23:45:00.579213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.577018 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkkkc\" (UniqueName: \"kubernetes.io/projected/da38ccbd-b489-4225-a4d1-4502943eb3d4-kube-api-access-kkkkc\") pod \"maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz\" (UID: \"da38ccbd-b489-4225-a4d1-4502943eb3d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" Oct 29 23:45:00.579213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.577063 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d322efb-9ed7-4868-b1b5-6ac26e0ce34d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh\" (UID: \"1d322efb-9ed7-4868-b1b5-6ac26e0ce34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" Oct 29 23:45:00.579213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.578143 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da38ccbd-b489-4225-a4d1-4502943eb3d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz\" (UID: \"da38ccbd-b489-4225-a4d1-4502943eb3d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" Oct 29 23:45:00.580481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.579710 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2707a1f-9602-475e-83ea-df56bcf01224-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5\" (UID: \"e2707a1f-9602-475e-83ea-df56bcf01224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" Oct 29 23:45:00.580481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.580450 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d322efb-9ed7-4868-b1b5-6ac26e0ce34d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh\" (UID: \"1d322efb-9ed7-4868-b1b5-6ac26e0ce34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" Oct 29 23:45:00.585828 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.584634 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" Oct 29 23:45:00.585828 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.585668 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8"] Oct 29 23:45:00.591261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.591241 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8" Oct 29 23:45:00.591816 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.591799 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" Oct 29 23:45:00.592675 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.592653 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv"] Oct 29 23:45:00.600436 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.600414 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8"] Oct 29 23:45:00.609949 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.608803 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkkkc\" (UniqueName: \"kubernetes.io/projected/da38ccbd-b489-4225-a4d1-4502943eb3d4-kube-api-access-kkkkc\") pod \"maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz\" (UID: \"da38ccbd-b489-4225-a4d1-4502943eb3d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" Oct 29 23:45:00.611395 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.611368 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zmgd\" (UniqueName: \"kubernetes.io/projected/e2707a1f-9602-475e-83ea-df56bcf01224-kube-api-access-5zmgd\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5\" (UID: \"e2707a1f-9602-475e-83ea-df56bcf01224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" Oct 29 23:45:00.615008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.614979 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2hhr\" (UniqueName: \"kubernetes.io/projected/1d322efb-9ed7-4868-b1b5-6ac26e0ce34d-kube-api-access-j2hhr\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh\" (UID: \"1d322efb-9ed7-4868-b1b5-6ac26e0ce34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" Oct 29 23:45:00.635601 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.635387 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6" Oct 29 23:45:00.650211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.647725 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" Oct 29 23:45:00.680243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.679026 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56gdd\" (UniqueName: \"kubernetes.io/projected/ca325758-625b-4f60-9b98-9a34cb9c35b7-kube-api-access-56gdd\") pod \"maas-api-key-cleanup-29888625-jpzwv\" (UID: \"ca325758-625b-4f60-9b98-9a34cb9c35b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" Oct 29 23:45:00.680243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.679212 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58tkn\" (UniqueName: \"kubernetes.io/projected/2c26d97b-29bd-4a99-813a-91cd764b3199-kube-api-access-58tkn\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5\" (UID: \"2c26d97b-29bd-4a99-813a-91cd764b3199\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" Oct 29 23:45:00.680243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.679251 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca325758-625b-4f60-9b98-9a34cb9c35b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888625-jpzwv\" (UID: \"ca325758-625b-4f60-9b98-9a34cb9c35b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" Oct 29 23:45:00.680243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.679312 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7k8g\" (UniqueName: \"kubernetes.io/projected/5d0cbcca-d158-4dbc-88f0-e827b54c630e-kube-api-access-g7k8g\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r\" (UID: \"5d0cbcca-d158-4dbc-88f0-e827b54c630e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" Oct 29 23:45:00.680243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.679358 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca7fdd3-eac8-49b9-bb6a-a621194a686b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj\" (UID: \"cca7fdd3-eac8-49b9-bb6a-a621194a686b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj" Oct 29 23:45:00.680243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.679415 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gct45\" (UniqueName: \"kubernetes.io/projected/2dc6fc01-4438-439e-8106-2a0c36a73f73-kube-api-access-gct45\") pod \"maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8\" (UID: \"2dc6fc01-4438-439e-8106-2a0c36a73f73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8" Oct 29 23:45:00.680243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.679538 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c26d97b-29bd-4a99-813a-91cd764b3199-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5\" (UID: \"2c26d97b-29bd-4a99-813a-91cd764b3199\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" Oct 29 23:45:00.680243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.679571 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkjqc\" (UniqueName: \"kubernetes.io/projected/f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142-kube-api-access-dkjqc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb\" (UID: \"f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb" Oct 29 23:45:00.680243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.679656 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv52m\" (UniqueName: \"kubernetes.io/projected/4d541e01-8a85-4dd3-801f-9a44798a0ed0-kube-api-access-tv52m\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b\" (UID: \"4d541e01-8a85-4dd3-801f-9a44798a0ed0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" Oct 29 23:45:00.680243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.679727 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb\" (UID: \"f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb" Oct 29 23:45:00.680243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.679814 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dc6fc01-4438-439e-8106-2a0c36a73f73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8\" (UID: \"2dc6fc01-4438-439e-8106-2a0c36a73f73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8" Oct 29 23:45:00.680243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.679933 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjbq4\" (UniqueName: \"kubernetes.io/projected/cca7fdd3-eac8-49b9-bb6a-a621194a686b-kube-api-access-sjbq4\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj\" (UID: \"cca7fdd3-eac8-49b9-bb6a-a621194a686b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj" Oct 29 23:45:00.680243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.679967 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d0cbcca-d158-4dbc-88f0-e827b54c630e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r\" (UID: \"5d0cbcca-d158-4dbc-88f0-e827b54c630e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" Oct 29 23:45:00.680243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.680012 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d541e01-8a85-4dd3-801f-9a44798a0ed0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b\" (UID: \"4d541e01-8a85-4dd3-801f-9a44798a0ed0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" Oct 29 23:45:00.696240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.681982 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c26d97b-29bd-4a99-813a-91cd764b3199-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5\" (UID: \"2c26d97b-29bd-4a99-813a-91cd764b3199\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" Oct 29 23:45:00.696240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.682885 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb\" (UID: \"f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb" Oct 29 23:45:00.776139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.771834 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58tkn\" (UniqueName: \"kubernetes.io/projected/2c26d97b-29bd-4a99-813a-91cd764b3199-kube-api-access-58tkn\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5\" (UID: \"2c26d97b-29bd-4a99-813a-91cd764b3199\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" Oct 29 23:45:00.777794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.777729 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkjqc\" (UniqueName: \"kubernetes.io/projected/f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142-kube-api-access-dkjqc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb\" (UID: \"f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb" Oct 29 23:45:00.787174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.783065 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca7fdd3-eac8-49b9-bb6a-a621194a686b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj\" (UID: \"cca7fdd3-eac8-49b9-bb6a-a621194a686b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj" Oct 29 23:45:00.787174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.783143 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gct45\" (UniqueName: \"kubernetes.io/projected/2dc6fc01-4438-439e-8106-2a0c36a73f73-kube-api-access-gct45\") pod \"maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8\" (UID: \"2dc6fc01-4438-439e-8106-2a0c36a73f73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8" Oct 29 23:45:00.787174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.784062 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv52m\" (UniqueName: \"kubernetes.io/projected/4d541e01-8a85-4dd3-801f-9a44798a0ed0-kube-api-access-tv52m\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b\" (UID: \"4d541e01-8a85-4dd3-801f-9a44798a0ed0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" Oct 29 23:45:00.787174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.784254 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dc6fc01-4438-439e-8106-2a0c36a73f73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8\" (UID: \"2dc6fc01-4438-439e-8106-2a0c36a73f73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8" Oct 29 23:45:00.787174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.784316 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca7fdd3-eac8-49b9-bb6a-a621194a686b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj\" (UID: \"cca7fdd3-eac8-49b9-bb6a-a621194a686b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj" Oct 29 23:45:00.787174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.784387 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjbq4\" (UniqueName: \"kubernetes.io/projected/cca7fdd3-eac8-49b9-bb6a-a621194a686b-kube-api-access-sjbq4\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj\" (UID: \"cca7fdd3-eac8-49b9-bb6a-a621194a686b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj" Oct 29 23:45:00.787174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.784428 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d0cbcca-d158-4dbc-88f0-e827b54c630e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r\" (UID: \"5d0cbcca-d158-4dbc-88f0-e827b54c630e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" Oct 29 23:45:00.787174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.784473 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d541e01-8a85-4dd3-801f-9a44798a0ed0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b\" (UID: \"4d541e01-8a85-4dd3-801f-9a44798a0ed0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" Oct 29 23:45:00.787174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.784538 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56gdd\" (UniqueName: \"kubernetes.io/projected/ca325758-625b-4f60-9b98-9a34cb9c35b7-kube-api-access-56gdd\") pod \"maas-api-key-cleanup-29888625-jpzwv\" (UID: \"ca325758-625b-4f60-9b98-9a34cb9c35b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" Oct 29 23:45:00.787174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.784624 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca325758-625b-4f60-9b98-9a34cb9c35b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888625-jpzwv\" (UID: \"ca325758-625b-4f60-9b98-9a34cb9c35b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" Oct 29 23:45:00.787174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.784681 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7k8g\" (UniqueName: \"kubernetes.io/projected/5d0cbcca-d158-4dbc-88f0-e827b54c630e-kube-api-access-g7k8g\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r\" (UID: \"5d0cbcca-d158-4dbc-88f0-e827b54c630e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" Oct 29 23:45:00.787174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.785078 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dc6fc01-4438-439e-8106-2a0c36a73f73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8\" (UID: \"2dc6fc01-4438-439e-8106-2a0c36a73f73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8" Oct 29 23:45:00.787174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.785832 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d0cbcca-d158-4dbc-88f0-e827b54c630e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r\" (UID: \"5d0cbcca-d158-4dbc-88f0-e827b54c630e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" Oct 29 23:45:00.787174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.786486 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d541e01-8a85-4dd3-801f-9a44798a0ed0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b\" (UID: \"4d541e01-8a85-4dd3-801f-9a44798a0ed0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" Oct 29 23:45:00.788460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.787713 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" Oct 29 23:45:00.790807 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.790764 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca325758-625b-4f60-9b98-9a34cb9c35b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888625-jpzwv\" (UID: \"ca325758-625b-4f60-9b98-9a34cb9c35b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" Oct 29 23:45:00.799774 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.797334 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" Oct 29 23:45:00.883675 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.883646 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjbq4\" (UniqueName: \"kubernetes.io/projected/cca7fdd3-eac8-49b9-bb6a-a621194a686b-kube-api-access-sjbq4\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj\" (UID: \"cca7fdd3-eac8-49b9-bb6a-a621194a686b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj" Oct 29 23:45:00.884625 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.884546 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv52m\" (UniqueName: \"kubernetes.io/projected/4d541e01-8a85-4dd3-801f-9a44798a0ed0-kube-api-access-tv52m\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b\" (UID: \"4d541e01-8a85-4dd3-801f-9a44798a0ed0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" Oct 29 23:45:00.884625 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.884588 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7k8g\" (UniqueName: \"kubernetes.io/projected/5d0cbcca-d158-4dbc-88f0-e827b54c630e-kube-api-access-g7k8g\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r\" (UID: \"5d0cbcca-d158-4dbc-88f0-e827b54c630e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" Oct 29 23:45:00.886243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.885247 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gct45\" (UniqueName: \"kubernetes.io/projected/2dc6fc01-4438-439e-8106-2a0c36a73f73-kube-api-access-gct45\") pod \"maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8\" (UID: \"2dc6fc01-4438-439e-8106-2a0c36a73f73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8" Oct 29 23:45:00.888409 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.888384 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56gdd\" (UniqueName: \"kubernetes.io/projected/ca325758-625b-4f60-9b98-9a34cb9c35b7-kube-api-access-56gdd\") pod \"maas-api-key-cleanup-29888625-jpzwv\" (UID: \"ca325758-625b-4f60-9b98-9a34cb9c35b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" Oct 29 23:45:00.948628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.948592 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb" Oct 29 23:45:00.985684 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:00.985649 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" Oct 29 23:45:01.007497 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.007272 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" Oct 29 23:45:01.027291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.025619 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" Oct 29 23:45:01.048409 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.043527 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj" Oct 29 23:45:01.077027 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.076663 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" Oct 29 23:45:01.102230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.102004 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8" Oct 29 23:45:01.104467 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.104350 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc"] Oct 29 23:45:01.179325 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.159229 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z"] Oct 29 23:45:01.296405 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.296363 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6"] Oct 29 23:45:01.301373 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:45:01.301085 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8496326e_aae6_49fc_be66_4cdeb78ccf74.slice/crio-eb0f828118bff65bf06385a6c2165bb4be10e952ff6e8ed69648d11e2e9b26de WatchSource:0}: Error finding container eb0f828118bff65bf06385a6c2165bb4be10e952ff6e8ed69648d11e2e9b26de: Status 404 returned error can't find the container with id eb0f828118bff65bf06385a6c2165bb4be10e952ff6e8ed69648d11e2e9b26de Oct 29 23:45:01.308827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.307954 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz"] Oct 29 23:45:01.324721 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:45:01.324499 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda38ccbd_b489_4225_a4d1_4502943eb3d4.slice/crio-d957f4b076da0cd4f302cfbeec5db65d09d12242760a0b5d04702124520c7015 WatchSource:0}: Error finding container d957f4b076da0cd4f302cfbeec5db65d09d12242760a0b5d04702124520c7015: Status 404 returned error can't find the container with id d957f4b076da0cd4f302cfbeec5db65d09d12242760a0b5d04702124520c7015 Oct 29 23:45:01.441705 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.438363 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5"] Oct 29 23:45:01.496885 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.496846 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="236cd687-b020-40e9-a69d-7acfcfe2b658" path="/var/lib/kubelet/pods/236cd687-b020-40e9-a69d-7acfcfe2b658/volumes" Oct 29 23:45:01.561098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.561065 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5"] Oct 29 23:45:01.568904 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.568351 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5"] Oct 29 23:45:01.654747 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.654712 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh"] Oct 29 23:45:01.702915 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.702887 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv"] Oct 29 23:45:01.714506 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:45:01.714479 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca325758_625b_4f60_9b98_9a34cb9c35b7.slice/crio-03614fe698382afcdd516f509a0081126a0f2b51eb429818c0ebfb92fef81be8 WatchSource:0}: Error finding container 03614fe698382afcdd516f509a0081126a0f2b51eb429818c0ebfb92fef81be8: Status 404 returned error can't find the container with id 03614fe698382afcdd516f509a0081126a0f2b51eb429818c0ebfb92fef81be8 Oct 29 23:45:01.715205 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:45:01.715096 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca7fdd3_eac8_49b9_bb6a_a621194a686b.slice/crio-138f98c167aa2763bedc8db1c2a0bdc1f1cc8c295c909240eeac891133369588 WatchSource:0}: Error finding container 138f98c167aa2763bedc8db1c2a0bdc1f1cc8c295c909240eeac891133369588: Status 404 returned error can't find the container with id 138f98c167aa2763bedc8db1c2a0bdc1f1cc8c295c909240eeac891133369588 Oct 29 23:45:01.716209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.716166 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj"] Oct 29 23:45:01.734574 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.733752 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb"] Oct 29 23:45:01.744204 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.744108 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5"] Oct 29 23:45:01.749241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.749217 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r"] Oct 29 23:45:01.749906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.749876 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" event={"ID":"1d322efb-9ed7-4868-b1b5-6ac26e0ce34d","Type":"ContainerStarted","Data":"d4979cb52d06f7ae50d2f4072e205c41cf12cfd6403522e039c729ab26b758b9"} Oct 29 23:45:01.751416 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.751389 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj" event={"ID":"cca7fdd3-eac8-49b9-bb6a-a621194a686b","Type":"ContainerStarted","Data":"138f98c167aa2763bedc8db1c2a0bdc1f1cc8c295c909240eeac891133369588"} Oct 29 23:45:01.753447 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.753419 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" event={"ID":"848ba138-a799-4dab-b704-2ae74e04ac95","Type":"ContainerStarted","Data":"7e34d8d69d1287bb4892ee3c4289aa56814e21b1dd64cfd9bd9bc1fcb23baca6"} Oct 29 23:45:01.755135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.755108 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc" event={"ID":"2b0ed631-9976-4f08-8d68-e783428c6a48","Type":"ContainerStarted","Data":"8c1218d62e44081e0fa82b7ba30e2cf93b58917925c1a12c4fa7ce48e2c970a7"} Oct 29 23:45:01.756923 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.756903 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" event={"ID":"da38ccbd-b489-4225-a4d1-4502943eb3d4","Type":"ContainerStarted","Data":"d957f4b076da0cd4f302cfbeec5db65d09d12242760a0b5d04702124520c7015"} Oct 29 23:45:01.759129 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.759103 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" event={"ID":"01fce6be-41f6-4874-b280-6f0422b87a06","Type":"ContainerStarted","Data":"0d5c7a0499d6b6edc5301d1da5b87abf971c99872a0c9721abc93f4a06b4c920"} Oct 29 23:45:01.761926 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.761901 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" event={"ID":"e2707a1f-9602-475e-83ea-df56bcf01224","Type":"ContainerStarted","Data":"9f8781f123aaf72e9078fe8761e09dfcf2966394bbcf4c5b15c151492a9c1a51"} Oct 29 23:45:01.763388 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.763365 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6" event={"ID":"8496326e-aae6-49fc-be66-4cdeb78ccf74","Type":"ContainerStarted","Data":"eb0f828118bff65bf06385a6c2165bb4be10e952ff6e8ed69648d11e2e9b26de"} Oct 29 23:45:01.764970 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.764951 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" event={"ID":"759dfe21-e08a-42ae-a63d-097ada97aa15","Type":"ContainerStarted","Data":"cc65a471f994385c4a2484c7c1826fbb65374abe7e5204cf5a7a07268f00bdba"} Oct 29 23:45:01.766452 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.766394 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" event={"ID":"ca325758-625b-4f60-9b98-9a34cb9c35b7","Type":"ContainerStarted","Data":"03614fe698382afcdd516f509a0081126a0f2b51eb429818c0ebfb92fef81be8"} Oct 29 23:45:01.776371 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.776348 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8"] Oct 29 23:45:01.865816 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:01.865789 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b"] Oct 29 23:45:02.776920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.776883 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" event={"ID":"4d541e01-8a85-4dd3-801f-9a44798a0ed0","Type":"ContainerStarted","Data":"e0f2a532ab1ad303e07631de98c55510aa962e19c6a3d38ba0ab5a2db7f5ca87"} Oct 29 23:45:02.778645 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.778618 5057 generic.go:356] "Generic (PLEG): container finished" podID="01fce6be-41f6-4874-b280-6f0422b87a06" containerID="cb59bbaca80f9a575dc30eaf0319f5bc9fd01bfd9e2f9bc3f2a8d11c2fc5a3e1" exitCode=60 Oct 29 23:45:02.778792 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.778653 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" event={"ID":"01fce6be-41f6-4874-b280-6f0422b87a06","Type":"ContainerDied","Data":"cb59bbaca80f9a575dc30eaf0319f5bc9fd01bfd9e2f9bc3f2a8d11c2fc5a3e1"} Oct 29 23:45:02.779069 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.779053 5057 scope.go:122] "RemoveContainer" containerID="cb59bbaca80f9a575dc30eaf0319f5bc9fd01bfd9e2f9bc3f2a8d11c2fc5a3e1" Oct 29 23:45:02.780323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.780291 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb" event={"ID":"f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142","Type":"ContainerStarted","Data":"094dba3686b0d2062de934a580ffb606f79b6403965af0b9c006b5b29f0ab04f"} Oct 29 23:45:02.782958 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.782929 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" event={"ID":"e2707a1f-9602-475e-83ea-df56bcf01224","Type":"ContainerStarted","Data":"d66fe6cb55ddad57caa346a50b4704479994ac324771fb364661ea2a5040af58"} Oct 29 23:45:02.784870 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.784846 5057 generic.go:356] "Generic (PLEG): container finished" podID="8496326e-aae6-49fc-be66-4cdeb78ccf74" containerID="7b92a8860316196906fca395d9a0bc4869628f680dda1b333eed67084dfa3620" exitCode=60 Oct 29 23:45:02.784976 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.784881 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6" event={"ID":"8496326e-aae6-49fc-be66-4cdeb78ccf74","Type":"ContainerDied","Data":"7b92a8860316196906fca395d9a0bc4869628f680dda1b333eed67084dfa3620"} Oct 29 23:45:02.785161 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.785144 5057 scope.go:122] "RemoveContainer" containerID="7b92a8860316196906fca395d9a0bc4869628f680dda1b333eed67084dfa3620" Oct 29 23:45:02.786789 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.786763 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" event={"ID":"759dfe21-e08a-42ae-a63d-097ada97aa15","Type":"ContainerStarted","Data":"de2fc5c05268dbb2c6b5dcde7573b91dac97f0c027b5a8b82a5cc75a894ef9c7"} Oct 29 23:45:02.788603 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.788550 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" event={"ID":"ca325758-625b-4f60-9b98-9a34cb9c35b7","Type":"ContainerStarted","Data":"b46331a878a57c4606a57366a248b205a59dda9b0918ae052b8dee23cc8ccc8e"} Oct 29 23:45:02.790693 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.790646 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" event={"ID":"1d322efb-9ed7-4868-b1b5-6ac26e0ce34d","Type":"ContainerStarted","Data":"82f0cebb70be9646dfa3c6f6afe36d36f64e0fb3fa6520052c0135befd9387a3"} Oct 29 23:45:02.792628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.792605 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" event={"ID":"848ba138-a799-4dab-b704-2ae74e04ac95","Type":"ContainerStarted","Data":"9150dade029e6134661ac98afb5f55743afd8660e2d9410fa47ec269a878d906"} Oct 29 23:45:02.794032 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.794002 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" event={"ID":"2c26d97b-29bd-4a99-813a-91cd764b3199","Type":"ContainerStarted","Data":"3a13f84792c6113f8e88f95153b5024b0fee3f3c8ebfa58fdf2d2cd93ca640fb"} Oct 29 23:45:02.796553 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.796523 5057 generic.go:356] "Generic (PLEG): container finished" podID="2b0ed631-9976-4f08-8d68-e783428c6a48" containerID="ac22446ac5154145c0a5cde0281362cad35069963a3364010a598289670f30bf" exitCode=60 Oct 29 23:45:02.796646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.796594 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc" event={"ID":"2b0ed631-9976-4f08-8d68-e783428c6a48","Type":"ContainerDied","Data":"ac22446ac5154145c0a5cde0281362cad35069963a3364010a598289670f30bf"} Oct 29 23:45:02.796825 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.796811 5057 scope.go:122] "RemoveContainer" containerID="ac22446ac5154145c0a5cde0281362cad35069963a3364010a598289670f30bf" Oct 29 23:45:02.798022 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.797999 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" event={"ID":"5d0cbcca-d158-4dbc-88f0-e827b54c630e","Type":"ContainerStarted","Data":"b591fc659eee8738a3d7789c81ea3b1064415aae4974e1d58fb173c663cf0311"} Oct 29 23:45:02.799418 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.799400 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8" event={"ID":"2dc6fc01-4438-439e-8106-2a0c36a73f73","Type":"ContainerStarted","Data":"83dc77f2c45cbb75b6ad3ae0fa57c3d0613f6c51269a5082db5b2a55f77507d1"} Oct 29 23:45:02.800926 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.800908 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" event={"ID":"da38ccbd-b489-4225-a4d1-4502943eb3d4","Type":"ContainerStarted","Data":"891e0ef6b0aeba3a864796168defe03d069b090634371b60e787bc4456c08efc"} Oct 29 23:45:02.878136 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.878038 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" podStartSLOduration=2.878017758 podStartE2EDuration="2.878017758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:02.868391603 +0000 UTC m=+3749.519890128" watchObservedRunningTime="2026-10-29 23:45:02.878017758 +0000 UTC m=+3749.529516292" Oct 29 23:45:02.995032 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.994981 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" podStartSLOduration=2.994966617 podStartE2EDuration="2.994966617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:02.992403691 +0000 UTC m=+3749.643902222" watchObservedRunningTime="2026-10-29 23:45:02.994966617 +0000 UTC m=+3749.646465168" Oct 29 23:45:02.995838 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:02.995779 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" podStartSLOduration=2.995755883 podStartE2EDuration="2.995755883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:02.953704498 +0000 UTC m=+3749.605203041" watchObservedRunningTime="2026-10-29 23:45:02.995755883 +0000 UTC m=+3749.647256856" Oct 29 23:45:03.033017 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.032959 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" podStartSLOduration=3.032942006 podStartE2EDuration="3.032942006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:03.030969656 +0000 UTC m=+3749.682468187" watchObservedRunningTime="2026-10-29 23:45:03.032942006 +0000 UTC m=+3749.684440537" Oct 29 23:45:03.075483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.075430 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" podStartSLOduration=3.0754144820000002 podStartE2EDuration="3.075414482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:03.069726286 +0000 UTC m=+3749.721224817" watchObservedRunningTime="2026-10-29 23:45:03.075414482 +0000 UTC m=+3749.726913042" Oct 29 23:45:03.151928 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.151827 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" podStartSLOduration=3.151810556 podStartE2EDuration="3.151810556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:03.149765154 +0000 UTC m=+3749.801263684" watchObservedRunningTime="2026-10-29 23:45:03.151810556 +0000 UTC m=+3749.803309085" Oct 29 23:45:03.423002 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.422972 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:45:03.756140 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:03.756092 5057 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b4ab25_72c1_4da3_bbdf_fb1e3ea7f142.slice/crio-conmon-4d374484b8bbefc3bbabcc02b1e21937da71437877af0ef04c86f26bbc0e071c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0cbcca_d158_4dbc_88f0_e827b54c630e.slice/crio-conmon-570517e99903eb8b44b44615d92147f0674462df3f3fa5d5648dd7254723b69b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca7fdd3_eac8_49b9_bb6a_a621194a686b.slice/crio-conmon-e50d71cafcb659cd28c748e2cfbf6f0fdf410d47cf826836d4412ceeb40f1785.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c26d97b_29bd_4a99_813a_91cd764b3199.slice/crio-9d0335624dcc4d853c3a2adc4540ed5fb683928e9e0dc3309262980405d6493f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0cbcca_d158_4dbc_88f0_e827b54c630e.slice/crio-570517e99903eb8b44b44615d92147f0674462df3f3fa5d5648dd7254723b69b.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:45:03.813334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.813303 5057 generic.go:356] "Generic (PLEG): container finished" podID="cca7fdd3-eac8-49b9-bb6a-a621194a686b" containerID="e50d71cafcb659cd28c748e2cfbf6f0fdf410d47cf826836d4412ceeb40f1785" exitCode=60 Oct 29 23:45:03.813809 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.813389 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj" event={"ID":"cca7fdd3-eac8-49b9-bb6a-a621194a686b","Type":"ContainerDied","Data":"e50d71cafcb659cd28c748e2cfbf6f0fdf410d47cf826836d4412ceeb40f1785"} Oct 29 23:45:03.813809 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.813738 5057 scope.go:122] "RemoveContainer" containerID="e50d71cafcb659cd28c748e2cfbf6f0fdf410d47cf826836d4412ceeb40f1785" Oct 29 23:45:03.815428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.815403 5057 generic.go:356] "Generic (PLEG): container finished" podID="848ba138-a799-4dab-b704-2ae74e04ac95" containerID="9150dade029e6134661ac98afb5f55743afd8660e2d9410fa47ec269a878d906" exitCode=60 Oct 29 23:45:03.815551 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.815484 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" event={"ID":"848ba138-a799-4dab-b704-2ae74e04ac95","Type":"ContainerDied","Data":"9150dade029e6134661ac98afb5f55743afd8660e2d9410fa47ec269a878d906"} Oct 29 23:45:03.815815 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.815798 5057 scope.go:122] "RemoveContainer" containerID="9150dade029e6134661ac98afb5f55743afd8660e2d9410fa47ec269a878d906" Oct 29 23:45:03.818063 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.818037 5057 generic.go:356] "Generic (PLEG): container finished" podID="2c26d97b-29bd-4a99-813a-91cd764b3199" containerID="9d0335624dcc4d853c3a2adc4540ed5fb683928e9e0dc3309262980405d6493f" exitCode=60 Oct 29 23:45:03.818162 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.818137 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" event={"ID":"2c26d97b-29bd-4a99-813a-91cd764b3199","Type":"ContainerDied","Data":"9d0335624dcc4d853c3a2adc4540ed5fb683928e9e0dc3309262980405d6493f"} Oct 29 23:45:03.818469 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.818449 5057 scope.go:122] "RemoveContainer" containerID="9d0335624dcc4d853c3a2adc4540ed5fb683928e9e0dc3309262980405d6493f" Oct 29 23:45:03.820053 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.820033 5057 generic.go:356] "Generic (PLEG): container finished" podID="5d0cbcca-d158-4dbc-88f0-e827b54c630e" containerID="570517e99903eb8b44b44615d92147f0674462df3f3fa5d5648dd7254723b69b" exitCode=60 Oct 29 23:45:03.820166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.820080 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" event={"ID":"5d0cbcca-d158-4dbc-88f0-e827b54c630e","Type":"ContainerDied","Data":"570517e99903eb8b44b44615d92147f0674462df3f3fa5d5648dd7254723b69b"} Oct 29 23:45:03.820474 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.820358 5057 scope.go:122] "RemoveContainer" containerID="570517e99903eb8b44b44615d92147f0674462df3f3fa5d5648dd7254723b69b" Oct 29 23:45:03.822225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.822204 5057 generic.go:356] "Generic (PLEG): container finished" podID="2dc6fc01-4438-439e-8106-2a0c36a73f73" containerID="c8f2c50fdda79f67d0e7132f2255c280a5768d93cdfa06a5740e706645d37cb2" exitCode=60 Oct 29 23:45:03.822311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.822283 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8" event={"ID":"2dc6fc01-4438-439e-8106-2a0c36a73f73","Type":"ContainerDied","Data":"c8f2c50fdda79f67d0e7132f2255c280a5768d93cdfa06a5740e706645d37cb2"} Oct 29 23:45:03.822513 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.822496 5057 scope.go:122] "RemoveContainer" containerID="c8f2c50fdda79f67d0e7132f2255c280a5768d93cdfa06a5740e706645d37cb2" Oct 29 23:45:03.824533 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.824511 5057 generic.go:356] "Generic (PLEG): container finished" podID="da38ccbd-b489-4225-a4d1-4502943eb3d4" containerID="891e0ef6b0aeba3a864796168defe03d069b090634371b60e787bc4456c08efc" exitCode=60 Oct 29 23:45:03.824654 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.824586 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" event={"ID":"da38ccbd-b489-4225-a4d1-4502943eb3d4","Type":"ContainerDied","Data":"891e0ef6b0aeba3a864796168defe03d069b090634371b60e787bc4456c08efc"} Oct 29 23:45:03.824795 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.824777 5057 scope.go:122] "RemoveContainer" containerID="891e0ef6b0aeba3a864796168defe03d069b090634371b60e787bc4456c08efc" Oct 29 23:45:03.826741 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.826720 5057 generic.go:356] "Generic (PLEG): container finished" podID="f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142" containerID="4d374484b8bbefc3bbabcc02b1e21937da71437877af0ef04c86f26bbc0e071c" exitCode=60 Oct 29 23:45:03.826851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.826743 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb" event={"ID":"f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142","Type":"ContainerDied","Data":"4d374484b8bbefc3bbabcc02b1e21937da71437877af0ef04c86f26bbc0e071c"} Oct 29 23:45:03.827024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.827004 5057 scope.go:122] "RemoveContainer" containerID="4d374484b8bbefc3bbabcc02b1e21937da71437877af0ef04c86f26bbc0e071c" Oct 29 23:45:03.830104 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.830084 5057 generic.go:356] "Generic (PLEG): container finished" podID="e2707a1f-9602-475e-83ea-df56bcf01224" containerID="d66fe6cb55ddad57caa346a50b4704479994ac324771fb364661ea2a5040af58" exitCode=60 Oct 29 23:45:03.830245 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.830226 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" event={"ID":"e2707a1f-9602-475e-83ea-df56bcf01224","Type":"ContainerDied","Data":"d66fe6cb55ddad57caa346a50b4704479994ac324771fb364661ea2a5040af58"} Oct 29 23:45:03.831008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.830570 5057 scope.go:122] "RemoveContainer" containerID="d66fe6cb55ddad57caa346a50b4704479994ac324771fb364661ea2a5040af58" Oct 29 23:45:03.834248 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.833795 5057 generic.go:356] "Generic (PLEG): container finished" podID="759dfe21-e08a-42ae-a63d-097ada97aa15" containerID="de2fc5c05268dbb2c6b5dcde7573b91dac97f0c027b5a8b82a5cc75a894ef9c7" exitCode=60 Oct 29 23:45:03.834248 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.833876 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" event={"ID":"759dfe21-e08a-42ae-a63d-097ada97aa15","Type":"ContainerDied","Data":"de2fc5c05268dbb2c6b5dcde7573b91dac97f0c027b5a8b82a5cc75a894ef9c7"} Oct 29 23:45:03.834248 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.834092 5057 scope.go:122] "RemoveContainer" containerID="de2fc5c05268dbb2c6b5dcde7573b91dac97f0c027b5a8b82a5cc75a894ef9c7" Oct 29 23:45:03.836000 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.835978 5057 generic.go:356] "Generic (PLEG): container finished" podID="ca325758-625b-4f60-9b98-9a34cb9c35b7" containerID="b46331a878a57c4606a57366a248b205a59dda9b0918ae052b8dee23cc8ccc8e" exitCode=60 Oct 29 23:45:03.836089 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.836059 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" event={"ID":"ca325758-625b-4f60-9b98-9a34cb9c35b7","Type":"ContainerDied","Data":"b46331a878a57c4606a57366a248b205a59dda9b0918ae052b8dee23cc8ccc8e"} Oct 29 23:45:03.836425 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.836396 5057 scope.go:122] "RemoveContainer" containerID="b46331a878a57c4606a57366a248b205a59dda9b0918ae052b8dee23cc8ccc8e" Oct 29 23:45:03.838222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.838182 5057 generic.go:356] "Generic (PLEG): container finished" podID="1d322efb-9ed7-4868-b1b5-6ac26e0ce34d" containerID="82f0cebb70be9646dfa3c6f6afe36d36f64e0fb3fa6520052c0135befd9387a3" exitCode=60 Oct 29 23:45:03.838294 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.838223 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" event={"ID":"1d322efb-9ed7-4868-b1b5-6ac26e0ce34d","Type":"ContainerDied","Data":"82f0cebb70be9646dfa3c6f6afe36d36f64e0fb3fa6520052c0135befd9387a3"} Oct 29 23:45:03.838530 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:03.838514 5057 scope.go:122] "RemoveContainer" containerID="82f0cebb70be9646dfa3c6f6afe36d36f64e0fb3fa6520052c0135befd9387a3" Oct 29 23:45:03.936209 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:03.936154 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:45:04.856813 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:04.856775 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6" event={"ID":"8496326e-aae6-49fc-be66-4cdeb78ccf74","Type":"ContainerStarted","Data":"8ab8fb0c28d08f9f71afe6be15ea184c2644e6b3bb5ff1b782761aa571d37bb8"} Oct 29 23:45:04.858998 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:04.858968 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" event={"ID":"ca325758-625b-4f60-9b98-9a34cb9c35b7","Type":"ContainerStarted","Data":"54dc9624534f2c3e30edb85a6d99b9beae2f30cee4fba59cae98852605439b76"} Oct 29 23:45:04.861125 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:04.861101 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" event={"ID":"2c26d97b-29bd-4a99-813a-91cd764b3199","Type":"ContainerStarted","Data":"5dbd9c21aa2b9be412d28c5c5f449ed18449495ef510ff61fd7f8d3a5676e9b0"} Oct 29 23:45:04.863277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:04.863247 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc" event={"ID":"2b0ed631-9976-4f08-8d68-e783428c6a48","Type":"ContainerStarted","Data":"fb700a84a96f8ad6c29f52339e3499900070a062c7d977d06a429792019ad62b"} Oct 29 23:45:04.865108 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:04.865084 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" event={"ID":"5d0cbcca-d158-4dbc-88f0-e827b54c630e","Type":"ContainerStarted","Data":"eeb7c616d0f62610e21e2f0b294fd5961b6ccea55e62f727709752b9094d1e17"} Oct 29 23:45:04.867384 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:04.867358 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8" event={"ID":"2dc6fc01-4438-439e-8106-2a0c36a73f73","Type":"ContainerStarted","Data":"b49dddb6d097010bb5518b938ba1d3224f5af4bdb82419c53824e1a6cbf69fd8"} Oct 29 23:45:04.869534 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:04.869512 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" event={"ID":"da38ccbd-b489-4225-a4d1-4502943eb3d4","Type":"ContainerStarted","Data":"49b3ce504664c7cecc01749771a852db2d358f76dc52dc053d39ace690c1d383"} Oct 29 23:45:04.871361 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:04.871334 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" event={"ID":"4d541e01-8a85-4dd3-801f-9a44798a0ed0","Type":"ContainerStarted","Data":"f3291bd5085407299540436c2f1b7b8031870c47c9f97791f40c424e676b2509"} Oct 29 23:45:04.873281 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:04.873257 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" event={"ID":"01fce6be-41f6-4874-b280-6f0422b87a06","Type":"ContainerStarted","Data":"fed549c7efff5e53518a4aa9646d8622d5aa6050b927bb67e5648a2e38acfd18"} Oct 29 23:45:04.900647 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:04.900544 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6" podStartSLOduration=4.900531154 podStartE2EDuration="4.900531154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:04.898025428 +0000 UTC m=+3751.549523962" watchObservedRunningTime="2026-10-29 23:45:04.900531154 +0000 UTC m=+3751.552029684" Oct 29 23:45:05.253463 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:05.251874 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" podStartSLOduration=5.251851967 podStartE2EDuration="5.251851967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:04.996686839 +0000 UTC m=+3751.648185371" watchObservedRunningTime="2026-10-29 23:45:05.251851967 +0000 UTC m=+3751.903350502" Oct 29 23:45:05.468209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:05.459960 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" podStartSLOduration=5.459939446 podStartE2EDuration="5.459939446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:05.060836984 +0000 UTC m=+3751.712335514" watchObservedRunningTime="2026-10-29 23:45:05.459939446 +0000 UTC m=+3752.111437977" Oct 29 23:45:05.807673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:05.807427 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8" podStartSLOduration=5.807406555 podStartE2EDuration="5.807406555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:05.108034814 +0000 UTC m=+3751.759533369" watchObservedRunningTime="2026-10-29 23:45:05.807406555 +0000 UTC m=+3752.458905087" Oct 29 23:45:05.811430 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:05.811395 5057 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca7fdd3_eac8_49b9_bb6a_a621194a686b.slice/crio-conmon-670122541b237b55e3fa938fb12a53ef98905b273dcaad28d943017308adcdf7.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:45:05.823888 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:05.823828 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" podStartSLOduration=5.82381013 podStartE2EDuration="5.82381013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:05.230599919 +0000 UTC m=+3751.882098455" watchObservedRunningTime="2026-10-29 23:45:05.82381013 +0000 UTC m=+3752.475308660" Oct 29 23:45:05.834438 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:05.834381 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" podStartSLOduration=5.83436368 podStartE2EDuration="5.83436368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:05.290633913 +0000 UTC m=+3751.942132445" watchObservedRunningTime="2026-10-29 23:45:05.83436368 +0000 UTC m=+3752.485862212" Oct 29 23:45:05.850974 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:05.848838 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc" podStartSLOduration=5.84881841 podStartE2EDuration="5.84881841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:05.339953162 +0000 UTC m=+3751.991451695" watchObservedRunningTime="2026-10-29 23:45:05.84881841 +0000 UTC m=+3752.500316945" Oct 29 23:45:05.911089 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:05.890174 5057 generic.go:356] "Generic (PLEG): container finished" podID="2b0ed631-9976-4f08-8d68-e783428c6a48" containerID="fb700a84a96f8ad6c29f52339e3499900070a062c7d977d06a429792019ad62b" exitCode=60 Oct 29 23:45:05.911089 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:05.890282 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc" event={"ID":"2b0ed631-9976-4f08-8d68-e783428c6a48","Type":"ContainerDied","Data":"fb700a84a96f8ad6c29f52339e3499900070a062c7d977d06a429792019ad62b"} Oct 29 23:45:05.911089 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:05.890344 5057 scope.go:122] "RemoveContainer" containerID="ac22446ac5154145c0a5cde0281362cad35069963a3364010a598289670f30bf" Oct 29 23:45:05.933320 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:05.912838 5057 scope.go:122] "RemoveContainer" containerID="fb700a84a96f8ad6c29f52339e3499900070a062c7d977d06a429792019ad62b" Oct 29 23:45:05.933320 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:05.913431 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc_odh-ai-gateway-infra(2b0ed631-9976-4f08-8d68-e783428c6a48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc" podUID="2b0ed631-9976-4f08-8d68-e783428c6a48" Oct 29 23:45:05.989876 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:05.973645 5057 generic.go:356] "Generic (PLEG): container finished" podID="5d0cbcca-d158-4dbc-88f0-e827b54c630e" containerID="eeb7c616d0f62610e21e2f0b294fd5961b6ccea55e62f727709752b9094d1e17" exitCode=60 Oct 29 23:45:05.989876 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:05.973872 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" event={"ID":"5d0cbcca-d158-4dbc-88f0-e827b54c630e","Type":"ContainerDied","Data":"eeb7c616d0f62610e21e2f0b294fd5961b6ccea55e62f727709752b9094d1e17"} Oct 29 23:45:06.027250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.020700 5057 scope.go:122] "RemoveContainer" containerID="eeb7c616d0f62610e21e2f0b294fd5961b6ccea55e62f727709752b9094d1e17" Oct 29 23:45:06.027250 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:06.021299 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r_odh-ai-gateway-infra(5d0cbcca-d158-4dbc-88f0-e827b54c630e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" podUID="5d0cbcca-d158-4dbc-88f0-e827b54c630e" Oct 29 23:45:06.051213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.030887 5057 generic.go:356] "Generic (PLEG): container finished" podID="2dc6fc01-4438-439e-8106-2a0c36a73f73" containerID="b49dddb6d097010bb5518b938ba1d3224f5af4bdb82419c53824e1a6cbf69fd8" exitCode=60 Oct 29 23:45:06.051213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.031019 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8" event={"ID":"2dc6fc01-4438-439e-8106-2a0c36a73f73","Type":"ContainerDied","Data":"b49dddb6d097010bb5518b938ba1d3224f5af4bdb82419c53824e1a6cbf69fd8"} Oct 29 23:45:06.051213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.031697 5057 scope.go:122] "RemoveContainer" containerID="b49dddb6d097010bb5518b938ba1d3224f5af4bdb82419c53824e1a6cbf69fd8" Oct 29 23:45:06.051213 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:06.032208 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8_odh-ai-gateway-infra(2dc6fc01-4438-439e-8106-2a0c36a73f73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8" podUID="2dc6fc01-4438-439e-8106-2a0c36a73f73" Oct 29 23:45:06.109211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.103758 5057 generic.go:356] "Generic (PLEG): container finished" podID="da38ccbd-b489-4225-a4d1-4502943eb3d4" containerID="49b3ce504664c7cecc01749771a852db2d358f76dc52dc053d39ace690c1d383" exitCode=60 Oct 29 23:45:06.109211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.103904 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" event={"ID":"da38ccbd-b489-4225-a4d1-4502943eb3d4","Type":"ContainerDied","Data":"49b3ce504664c7cecc01749771a852db2d358f76dc52dc053d39ace690c1d383"} Oct 29 23:45:06.109211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.104602 5057 scope.go:122] "RemoveContainer" containerID="49b3ce504664c7cecc01749771a852db2d358f76dc52dc053d39ace690c1d383" Oct 29 23:45:06.109211 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:06.105271 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz_odh-ai-gateway-infra(da38ccbd-b489-4225-a4d1-4502943eb3d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" podUID="da38ccbd-b489-4225-a4d1-4502943eb3d4" Oct 29 23:45:06.213952 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.213757 5057 generic.go:356] "Generic (PLEG): container finished" podID="4d541e01-8a85-4dd3-801f-9a44798a0ed0" containerID="f3291bd5085407299540436c2f1b7b8031870c47c9f97791f40c424e676b2509" exitCode=60 Oct 29 23:45:06.233828 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.233790 5057 generic.go:356] "Generic (PLEG): container finished" podID="01fce6be-41f6-4874-b280-6f0422b87a06" containerID="fed549c7efff5e53518a4aa9646d8622d5aa6050b927bb67e5648a2e38acfd18" exitCode=60 Oct 29 23:45:06.235515 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.235468 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" event={"ID":"4d541e01-8a85-4dd3-801f-9a44798a0ed0","Type":"ContainerDied","Data":"f3291bd5085407299540436c2f1b7b8031870c47c9f97791f40c424e676b2509"} Oct 29 23:45:06.235756 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.235737 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" event={"ID":"01fce6be-41f6-4874-b280-6f0422b87a06","Type":"ContainerDied","Data":"fed549c7efff5e53518a4aa9646d8622d5aa6050b927bb67e5648a2e38acfd18"} Oct 29 23:45:06.518210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.516588 5057 generic.go:356] "Generic (PLEG): container finished" podID="f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142" containerID="8447e37d6aeffa922849626901ada9bbf63bcf17e22edcb5b3933446d05654f1" exitCode=60 Oct 29 23:45:06.518210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.516756 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb" event={"ID":"f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142","Type":"ContainerDied","Data":"8447e37d6aeffa922849626901ada9bbf63bcf17e22edcb5b3933446d05654f1"} Oct 29 23:45:06.518477 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.518332 5057 scope.go:122] "RemoveContainer" containerID="f3291bd5085407299540436c2f1b7b8031870c47c9f97791f40c424e676b2509" Oct 29 23:45:06.785375 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.785329 5057 scope.go:122] "RemoveContainer" containerID="fed549c7efff5e53518a4aa9646d8622d5aa6050b927bb67e5648a2e38acfd18" Oct 29 23:45:06.789362 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:06.789315 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z_odh-ai-gateway-infra(01fce6be-41f6-4874-b280-6f0422b87a06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" podUID="01fce6be-41f6-4874-b280-6f0422b87a06" Oct 29 23:45:06.818703 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.814217 5057 scope.go:122] "RemoveContainer" containerID="570517e99903eb8b44b44615d92147f0674462df3f3fa5d5648dd7254723b69b" Oct 29 23:45:06.825093 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.825052 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" event={"ID":"e2707a1f-9602-475e-83ea-df56bcf01224","Type":"ContainerStarted","Data":"f9867c09d2c81255d094ba5162971ca7fb24f4ec52a16c7c5a35dde39fd7801a"} Oct 29 23:45:06.837996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.837959 5057 generic.go:356] "Generic (PLEG): container finished" podID="8496326e-aae6-49fc-be66-4cdeb78ccf74" containerID="8ab8fb0c28d08f9f71afe6be15ea184c2644e6b3bb5ff1b782761aa571d37bb8" exitCode=60 Oct 29 23:45:06.838409 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.838372 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6" event={"ID":"8496326e-aae6-49fc-be66-4cdeb78ccf74","Type":"ContainerDied","Data":"8ab8fb0c28d08f9f71afe6be15ea184c2644e6b3bb5ff1b782761aa571d37bb8"} Oct 29 23:45:06.839020 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.839000 5057 scope.go:122] "RemoveContainer" containerID="8447e37d6aeffa922849626901ada9bbf63bcf17e22edcb5b3933446d05654f1" Oct 29 23:45:06.839675 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:06.839635 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb_odh-ai-gateway-infra(f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb" podUID="f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142" Oct 29 23:45:06.842210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.840004 5057 scope.go:122] "RemoveContainer" containerID="f9867c09d2c81255d094ba5162971ca7fb24f4ec52a16c7c5a35dde39fd7801a" Oct 29 23:45:06.842210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.840284 5057 scope.go:122] "RemoveContainer" containerID="8ab8fb0c28d08f9f71afe6be15ea184c2644e6b3bb5ff1b782761aa571d37bb8" Oct 29 23:45:06.842210 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:06.840501 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5_odh-ai-gateway-infra(e2707a1f-9602-475e-83ea-df56bcf01224)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" podUID="e2707a1f-9602-475e-83ea-df56bcf01224" Oct 29 23:45:06.842210 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:06.840798 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6_odh-ai-gateway-infra(8496326e-aae6-49fc-be66-4cdeb78ccf74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6" podUID="8496326e-aae6-49fc-be66-4cdeb78ccf74" Oct 29 23:45:06.846428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.845614 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" event={"ID":"759dfe21-e08a-42ae-a63d-097ada97aa15","Type":"ContainerStarted","Data":"e2c29de89ed17785b9460425ab2c1cf9cda959e27ed6e7756d854ee85c1b6899"} Oct 29 23:45:06.846428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.846066 5057 scope.go:122] "RemoveContainer" containerID="e2c29de89ed17785b9460425ab2c1cf9cda959e27ed6e7756d854ee85c1b6899" Oct 29 23:45:06.846584 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:06.846534 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5_odh-ai-gateway-infra(759dfe21-e08a-42ae-a63d-097ada97aa15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" podUID="759dfe21-e08a-42ae-a63d-097ada97aa15" Oct 29 23:45:06.855200 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.854802 5057 generic.go:356] "Generic (PLEG): container finished" podID="ca325758-625b-4f60-9b98-9a34cb9c35b7" containerID="54dc9624534f2c3e30edb85a6d99b9beae2f30cee4fba59cae98852605439b76" exitCode=60 Oct 29 23:45:06.855409 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.855385 5057 scope.go:122] "RemoveContainer" containerID="54dc9624534f2c3e30edb85a6d99b9beae2f30cee4fba59cae98852605439b76" Oct 29 23:45:06.856515 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:06.856252 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888625-jpzwv_odh-ai-gateway-infra(ca325758-625b-4f60-9b98-9a34cb9c35b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" podUID="ca325758-625b-4f60-9b98-9a34cb9c35b7" Oct 29 23:45:06.857168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.855395 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" event={"ID":"ca325758-625b-4f60-9b98-9a34cb9c35b7","Type":"ContainerDied","Data":"54dc9624534f2c3e30edb85a6d99b9beae2f30cee4fba59cae98852605439b76"} Oct 29 23:45:06.863915 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.863886 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" event={"ID":"1d322efb-9ed7-4868-b1b5-6ac26e0ce34d","Type":"ContainerStarted","Data":"c5b4b74a4534e573560ccde8a9f510eaa9385ffa77a1711d5111f605da604ae6"} Oct 29 23:45:06.864747 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.864724 5057 scope.go:122] "RemoveContainer" containerID="c5b4b74a4534e573560ccde8a9f510eaa9385ffa77a1711d5111f605da604ae6" Oct 29 23:45:06.865677 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:06.865652 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh_odh-ai-gateway-infra(1d322efb-9ed7-4868-b1b5-6ac26e0ce34d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" podUID="1d322efb-9ed7-4868-b1b5-6ac26e0ce34d" Oct 29 23:45:06.868787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.868761 5057 generic.go:356] "Generic (PLEG): container finished" podID="cca7fdd3-eac8-49b9-bb6a-a621194a686b" containerID="670122541b237b55e3fa938fb12a53ef98905b273dcaad28d943017308adcdf7" exitCode=60 Oct 29 23:45:06.868889 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.868854 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj" event={"ID":"cca7fdd3-eac8-49b9-bb6a-a621194a686b","Type":"ContainerDied","Data":"670122541b237b55e3fa938fb12a53ef98905b273dcaad28d943017308adcdf7"} Oct 29 23:45:06.869344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.869305 5057 scope.go:122] "RemoveContainer" containerID="670122541b237b55e3fa938fb12a53ef98905b273dcaad28d943017308adcdf7" Oct 29 23:45:06.869948 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:06.869904 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj_odh-ai-gateway-infra(cca7fdd3-eac8-49b9-bb6a-a621194a686b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj" podUID="cca7fdd3-eac8-49b9-bb6a-a621194a686b" Oct 29 23:45:06.871920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.871886 5057 generic.go:356] "Generic (PLEG): container finished" podID="848ba138-a799-4dab-b704-2ae74e04ac95" containerID="5b644524e0bafc859fdf8b99e53ab162e6280eb4e1b3316a128eb1191fa9d545" exitCode=60 Oct 29 23:45:06.872045 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.872024 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" event={"ID":"848ba138-a799-4dab-b704-2ae74e04ac95","Type":"ContainerDied","Data":"5b644524e0bafc859fdf8b99e53ab162e6280eb4e1b3316a128eb1191fa9d545"} Oct 29 23:45:06.872668 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.872646 5057 scope.go:122] "RemoveContainer" containerID="5b644524e0bafc859fdf8b99e53ab162e6280eb4e1b3316a128eb1191fa9d545" Oct 29 23:45:06.873223 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:06.873198 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5_odh-ai-gateway-infra(848ba138-a799-4dab-b704-2ae74e04ac95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" podUID="848ba138-a799-4dab-b704-2ae74e04ac95" Oct 29 23:45:06.873317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.873245 5057 scope.go:122] "RemoveContainer" containerID="c8f2c50fdda79f67d0e7132f2255c280a5768d93cdfa06a5740e706645d37cb2" Oct 29 23:45:06.875295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.875275 5057 generic.go:356] "Generic (PLEG): container finished" podID="2c26d97b-29bd-4a99-813a-91cd764b3199" containerID="5dbd9c21aa2b9be412d28c5c5f449ed18449495ef510ff61fd7f8d3a5676e9b0" exitCode=60 Oct 29 23:45:06.875385 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.875314 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" event={"ID":"2c26d97b-29bd-4a99-813a-91cd764b3199","Type":"ContainerDied","Data":"5dbd9c21aa2b9be412d28c5c5f449ed18449495ef510ff61fd7f8d3a5676e9b0"} Oct 29 23:45:06.875808 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.875790 5057 scope.go:122] "RemoveContainer" containerID="5dbd9c21aa2b9be412d28c5c5f449ed18449495ef510ff61fd7f8d3a5676e9b0" Oct 29 23:45:06.876172 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:06.876150 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5_odh-ai-gateway-infra(2c26d97b-29bd-4a99-813a-91cd764b3199)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" podUID="2c26d97b-29bd-4a99-813a-91cd764b3199" Oct 29 23:45:06.910348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.910328 5057 scope.go:122] "RemoveContainer" containerID="891e0ef6b0aeba3a864796168defe03d069b090634371b60e787bc4456c08efc" Oct 29 23:45:06.927965 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.927941 5057 scope.go:122] "RemoveContainer" containerID="cb59bbaca80f9a575dc30eaf0319f5bc9fd01bfd9e2f9bc3f2a8d11c2fc5a3e1" Oct 29 23:45:06.944913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.944887 5057 scope.go:122] "RemoveContainer" containerID="4d374484b8bbefc3bbabcc02b1e21937da71437877af0ef04c86f26bbc0e071c" Oct 29 23:45:06.960694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.960669 5057 scope.go:122] "RemoveContainer" containerID="7b92a8860316196906fca395d9a0bc4869628f680dda1b333eed67084dfa3620" Oct 29 23:45:06.977769 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.977743 5057 scope.go:122] "RemoveContainer" containerID="b46331a878a57c4606a57366a248b205a59dda9b0918ae052b8dee23cc8ccc8e" Oct 29 23:45:06.997074 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:06.996895 5057 scope.go:122] "RemoveContainer" containerID="e50d71cafcb659cd28c748e2cfbf6f0fdf410d47cf826836d4412ceeb40f1785" Oct 29 23:45:07.016095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.016070 5057 scope.go:122] "RemoveContainer" containerID="9150dade029e6134661ac98afb5f55743afd8660e2d9410fa47ec269a878d906" Oct 29 23:45:07.036858 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.036794 5057 scope.go:122] "RemoveContainer" containerID="9d0335624dcc4d853c3a2adc4540ed5fb683928e9e0dc3309262980405d6493f" Oct 29 23:45:07.887470 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.887434 5057 scope.go:122] "RemoveContainer" containerID="fb700a84a96f8ad6c29f52339e3499900070a062c7d977d06a429792019ad62b" Oct 29 23:45:07.887908 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:07.887884 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc_odh-ai-gateway-infra(2b0ed631-9976-4f08-8d68-e783428c6a48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc" podUID="2b0ed631-9976-4f08-8d68-e783428c6a48" Oct 29 23:45:07.889422 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.889397 5057 scope.go:122] "RemoveContainer" containerID="eeb7c616d0f62610e21e2f0b294fd5961b6ccea55e62f727709752b9094d1e17" Oct 29 23:45:07.889828 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:07.889802 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r_odh-ai-gateway-infra(5d0cbcca-d158-4dbc-88f0-e827b54c630e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" podUID="5d0cbcca-d158-4dbc-88f0-e827b54c630e" Oct 29 23:45:07.891881 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.891863 5057 scope.go:122] "RemoveContainer" containerID="b49dddb6d097010bb5518b938ba1d3224f5af4bdb82419c53824e1a6cbf69fd8" Oct 29 23:45:07.892201 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:07.892169 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8_odh-ai-gateway-infra(2dc6fc01-4438-439e-8106-2a0c36a73f73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8" podUID="2dc6fc01-4438-439e-8106-2a0c36a73f73" Oct 29 23:45:07.893772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.893753 5057 scope.go:122] "RemoveContainer" containerID="49b3ce504664c7cecc01749771a852db2d358f76dc52dc053d39ace690c1d383" Oct 29 23:45:07.894097 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:07.894072 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz_odh-ai-gateway-infra(da38ccbd-b489-4225-a4d1-4502943eb3d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" podUID="da38ccbd-b489-4225-a4d1-4502943eb3d4" Oct 29 23:45:07.895573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.895550 5057 generic.go:356] "Generic (PLEG): container finished" podID="4d541e01-8a85-4dd3-801f-9a44798a0ed0" containerID="43a03ebf3235b729925abe2361c0f28452f94f42002ebecaf2d71311d26d1c83" exitCode=60 Oct 29 23:45:07.895662 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.895629 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" event={"ID":"4d541e01-8a85-4dd3-801f-9a44798a0ed0","Type":"ContainerDied","Data":"43a03ebf3235b729925abe2361c0f28452f94f42002ebecaf2d71311d26d1c83"} Oct 29 23:45:07.895728 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.895681 5057 scope.go:122] "RemoveContainer" containerID="f3291bd5085407299540436c2f1b7b8031870c47c9f97791f40c424e676b2509" Oct 29 23:45:07.895844 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.895829 5057 scope.go:122] "RemoveContainer" containerID="43a03ebf3235b729925abe2361c0f28452f94f42002ebecaf2d71311d26d1c83" Oct 29 23:45:07.896109 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:07.896082 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b_odh-ai-gateway-infra(4d541e01-8a85-4dd3-801f-9a44798a0ed0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" podUID="4d541e01-8a85-4dd3-801f-9a44798a0ed0" Oct 29 23:45:07.897810 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.897793 5057 scope.go:122] "RemoveContainer" containerID="fed549c7efff5e53518a4aa9646d8622d5aa6050b927bb67e5648a2e38acfd18" Oct 29 23:45:07.898060 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:07.898038 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z_odh-ai-gateway-infra(01fce6be-41f6-4874-b280-6f0422b87a06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" podUID="01fce6be-41f6-4874-b280-6f0422b87a06" Oct 29 23:45:07.899410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.899392 5057 scope.go:122] "RemoveContainer" containerID="8447e37d6aeffa922849626901ada9bbf63bcf17e22edcb5b3933446d05654f1" Oct 29 23:45:07.899669 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:07.899649 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb_odh-ai-gateway-infra(f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb" podUID="f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142" Oct 29 23:45:07.901338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.901310 5057 generic.go:356] "Generic (PLEG): container finished" podID="e2707a1f-9602-475e-83ea-df56bcf01224" containerID="f9867c09d2c81255d094ba5162971ca7fb24f4ec52a16c7c5a35dde39fd7801a" exitCode=60 Oct 29 23:45:07.901444 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.901341 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" event={"ID":"e2707a1f-9602-475e-83ea-df56bcf01224","Type":"ContainerDied","Data":"f9867c09d2c81255d094ba5162971ca7fb24f4ec52a16c7c5a35dde39fd7801a"} Oct 29 23:45:07.901686 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.901666 5057 scope.go:122] "RemoveContainer" containerID="f9867c09d2c81255d094ba5162971ca7fb24f4ec52a16c7c5a35dde39fd7801a" Oct 29 23:45:07.901974 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:07.901950 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5_odh-ai-gateway-infra(e2707a1f-9602-475e-83ea-df56bcf01224)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" podUID="e2707a1f-9602-475e-83ea-df56bcf01224" Oct 29 23:45:07.905705 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.905682 5057 scope.go:122] "RemoveContainer" containerID="8ab8fb0c28d08f9f71afe6be15ea184c2644e6b3bb5ff1b782761aa571d37bb8" Oct 29 23:45:07.905992 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:07.905970 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6_odh-ai-gateway-infra(8496326e-aae6-49fc-be66-4cdeb78ccf74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6" podUID="8496326e-aae6-49fc-be66-4cdeb78ccf74" Oct 29 23:45:07.908124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.908103 5057 generic.go:356] "Generic (PLEG): container finished" podID="759dfe21-e08a-42ae-a63d-097ada97aa15" containerID="e2c29de89ed17785b9460425ab2c1cf9cda959e27ed6e7756d854ee85c1b6899" exitCode=60 Oct 29 23:45:07.908344 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.908177 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" event={"ID":"759dfe21-e08a-42ae-a63d-097ada97aa15","Type":"ContainerDied","Data":"e2c29de89ed17785b9460425ab2c1cf9cda959e27ed6e7756d854ee85c1b6899"} Oct 29 23:45:07.908491 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.908465 5057 scope.go:122] "RemoveContainer" containerID="e2c29de89ed17785b9460425ab2c1cf9cda959e27ed6e7756d854ee85c1b6899" Oct 29 23:45:07.908781 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:07.908758 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5_odh-ai-gateway-infra(759dfe21-e08a-42ae-a63d-097ada97aa15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" podUID="759dfe21-e08a-42ae-a63d-097ada97aa15" Oct 29 23:45:07.910628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.910608 5057 scope.go:122] "RemoveContainer" containerID="54dc9624534f2c3e30edb85a6d99b9beae2f30cee4fba59cae98852605439b76" Oct 29 23:45:07.910954 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:07.910926 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888625-jpzwv_odh-ai-gateway-infra(ca325758-625b-4f60-9b98-9a34cb9c35b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" podUID="ca325758-625b-4f60-9b98-9a34cb9c35b7" Oct 29 23:45:07.913024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.912992 5057 generic.go:356] "Generic (PLEG): container finished" podID="1d322efb-9ed7-4868-b1b5-6ac26e0ce34d" containerID="c5b4b74a4534e573560ccde8a9f510eaa9385ffa77a1711d5111f605da604ae6" exitCode=60 Oct 29 23:45:07.913121 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.913099 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" event={"ID":"1d322efb-9ed7-4868-b1b5-6ac26e0ce34d","Type":"ContainerDied","Data":"c5b4b74a4534e573560ccde8a9f510eaa9385ffa77a1711d5111f605da604ae6"} Oct 29 23:45:07.913381 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.913360 5057 scope.go:122] "RemoveContainer" containerID="d66fe6cb55ddad57caa346a50b4704479994ac324771fb364661ea2a5040af58" Oct 29 23:45:07.913709 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.913688 5057 scope.go:122] "RemoveContainer" containerID="c5b4b74a4534e573560ccde8a9f510eaa9385ffa77a1711d5111f605da604ae6" Oct 29 23:45:07.914104 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:07.914069 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh_odh-ai-gateway-infra(1d322efb-9ed7-4868-b1b5-6ac26e0ce34d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" podUID="1d322efb-9ed7-4868-b1b5-6ac26e0ce34d" Oct 29 23:45:07.915443 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.915424 5057 scope.go:122] "RemoveContainer" containerID="670122541b237b55e3fa938fb12a53ef98905b273dcaad28d943017308adcdf7" Oct 29 23:45:07.915715 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:07.915693 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj_odh-ai-gateway-infra(cca7fdd3-eac8-49b9-bb6a-a621194a686b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj" podUID="cca7fdd3-eac8-49b9-bb6a-a621194a686b" Oct 29 23:45:07.917243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.917221 5057 scope.go:122] "RemoveContainer" containerID="5b644524e0bafc859fdf8b99e53ab162e6280eb4e1b3316a128eb1191fa9d545" Oct 29 23:45:07.917522 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:07.917499 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5_odh-ai-gateway-infra(848ba138-a799-4dab-b704-2ae74e04ac95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" podUID="848ba138-a799-4dab-b704-2ae74e04ac95" Oct 29 23:45:07.919076 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.919053 5057 scope.go:122] "RemoveContainer" containerID="5dbd9c21aa2b9be412d28c5c5f449ed18449495ef510ff61fd7f8d3a5676e9b0" Oct 29 23:45:07.919367 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:07.919347 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5_odh-ai-gateway-infra(2c26d97b-29bd-4a99-813a-91cd764b3199)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" podUID="2c26d97b-29bd-4a99-813a-91cd764b3199" Oct 29 23:45:07.931267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.931242 5057 scope.go:122] "RemoveContainer" containerID="de2fc5c05268dbb2c6b5dcde7573b91dac97f0c027b5a8b82a5cc75a894ef9c7" Oct 29 23:45:07.948473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:07.948450 5057 scope.go:122] "RemoveContainer" containerID="82f0cebb70be9646dfa3c6f6afe36d36f64e0fb3fa6520052c0135befd9387a3" Oct 29 23:45:08.932029 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:08.932000 5057 scope.go:122] "RemoveContainer" containerID="43a03ebf3235b729925abe2361c0f28452f94f42002ebecaf2d71311d26d1c83" Oct 29 23:45:08.932539 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:08.932451 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b_odh-ai-gateway-infra(4d541e01-8a85-4dd3-801f-9a44798a0ed0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" podUID="4d541e01-8a85-4dd3-801f-9a44798a0ed0" Oct 29 23:45:08.934033 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:08.934011 5057 scope.go:122] "RemoveContainer" containerID="f9867c09d2c81255d094ba5162971ca7fb24f4ec52a16c7c5a35dde39fd7801a" Oct 29 23:45:08.934460 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:08.934433 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5_odh-ai-gateway-infra(e2707a1f-9602-475e-83ea-df56bcf01224)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" podUID="e2707a1f-9602-475e-83ea-df56bcf01224" Oct 29 23:45:08.936257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:08.936238 5057 scope.go:122] "RemoveContainer" containerID="e2c29de89ed17785b9460425ab2c1cf9cda959e27ed6e7756d854ee85c1b6899" Oct 29 23:45:08.936593 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:08.936566 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5_odh-ai-gateway-infra(759dfe21-e08a-42ae-a63d-097ada97aa15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" podUID="759dfe21-e08a-42ae-a63d-097ada97aa15" Oct 29 23:45:08.938229 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:08.938209 5057 scope.go:122] "RemoveContainer" containerID="c5b4b74a4534e573560ccde8a9f510eaa9385ffa77a1711d5111f605da604ae6" Oct 29 23:45:08.938491 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:08.938472 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh_odh-ai-gateway-infra(1d322efb-9ed7-4868-b1b5-6ac26e0ce34d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" podUID="1d322efb-9ed7-4868-b1b5-6ac26e0ce34d" Oct 29 23:45:16.247522 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:16.247483 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6"] Oct 29 23:45:16.482211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:16.477590 5057 scope.go:122] "RemoveContainer" containerID="8447e37d6aeffa922849626901ada9bbf63bcf17e22edcb5b3933446d05654f1" Oct 29 23:45:16.482211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:16.478737 5057 scope.go:122] "RemoveContainer" containerID="fed549c7efff5e53518a4aa9646d8622d5aa6050b927bb67e5648a2e38acfd18" Oct 29 23:45:16.482211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:16.479587 5057 scope.go:122] "RemoveContainer" containerID="eeb7c616d0f62610e21e2f0b294fd5961b6ccea55e62f727709752b9094d1e17" Oct 29 23:45:16.584764 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:16.581784 5057 scope.go:122] "RemoveContainer" containerID="54dc9624534f2c3e30edb85a6d99b9beae2f30cee4fba59cae98852605439b76" Oct 29 23:45:16.584764 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:16.582396 5057 scope.go:122] "RemoveContainer" containerID="5dbd9c21aa2b9be412d28c5c5f449ed18449495ef510ff61fd7f8d3a5676e9b0" Oct 29 23:45:16.762364 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:16.762291 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6" Oct 29 23:45:16.975228 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:16.941030 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8496326e-aae6-49fc-be66-4cdeb78ccf74-kube-api-access-zlcw7\" (UniqueName: \"kubernetes.io/projected/8496326e-aae6-49fc-be66-4cdeb78ccf74-kube-api-access-zlcw7\") pod \"8496326e-aae6-49fc-be66-4cdeb78ccf74\" (UID: \"8496326e-aae6-49fc-be66-4cdeb78ccf74\") " Oct 29 23:45:16.975228 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:16.941431 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8496326e-aae6-49fc-be66-4cdeb78ccf74-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8496326e-aae6-49fc-be66-4cdeb78ccf74-maas-api-service-ca\") pod \"8496326e-aae6-49fc-be66-4cdeb78ccf74\" (UID: \"8496326e-aae6-49fc-be66-4cdeb78ccf74\") " Oct 29 23:45:16.978865 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:16.978817 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8496326e-aae6-49fc-be66-4cdeb78ccf74-maas-api-service-ca" pod "8496326e-aae6-49fc-be66-4cdeb78ccf74" (UID: "8496326e-aae6-49fc-be66-4cdeb78ccf74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:17.008243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:16.989610 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8496326e-aae6-49fc-be66-4cdeb78ccf74-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:17.042253 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:17.039443 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8496326e-aae6-49fc-be66-4cdeb78ccf74-kube-api-access-zlcw7" pod "8496326e-aae6-49fc-be66-4cdeb78ccf74" (UID: "8496326e-aae6-49fc-be66-4cdeb78ccf74"). InnerVolumeSpecName "kube-api-access-zlcw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:17.107849 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:17.106821 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlcw7\" (UniqueName: \"kubernetes.io/projected/8496326e-aae6-49fc-be66-4cdeb78ccf74-kube-api-access-zlcw7\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:17.124777 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:17.124150 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6" event={"ID":"8496326e-aae6-49fc-be66-4cdeb78ccf74","Type":"ContainerDied","Data":"eb0f828118bff65bf06385a6c2165bb4be10e952ff6e8ed69648d11e2e9b26de"} Oct 29 23:45:17.124777 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:17.124383 5057 scope.go:122] "RemoveContainer" containerID="8ab8fb0c28d08f9f71afe6be15ea184c2644e6b3bb5ff1b782761aa571d37bb8" Oct 29 23:45:17.124777 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:17.124748 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6" Oct 29 23:45:17.217040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:17.217002 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6"] Oct 29 23:45:17.230921 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:17.230835 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888625-t99d6"] Oct 29 23:45:17.489539 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:17.482950 5057 scope.go:122] "RemoveContainer" containerID="c5b4b74a4534e573560ccde8a9f510eaa9385ffa77a1711d5111f605da604ae6" Oct 29 23:45:17.489539 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:17.484959 5057 scope.go:122] "RemoveContainer" containerID="b49dddb6d097010bb5518b938ba1d3224f5af4bdb82419c53824e1a6cbf69fd8" Oct 29 23:45:17.489539 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:17.486069 5057 scope.go:122] "RemoveContainer" containerID="670122541b237b55e3fa938fb12a53ef98905b273dcaad28d943017308adcdf7" Oct 29 23:45:17.489539 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:17.488429 5057 scope.go:122] "RemoveContainer" containerID="f9867c09d2c81255d094ba5162971ca7fb24f4ec52a16c7c5a35dde39fd7801a" Oct 29 23:45:17.515893 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:17.515855 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8496326e-aae6-49fc-be66-4cdeb78ccf74" path="/var/lib/kubelet/pods/8496326e-aae6-49fc-be66-4cdeb78ccf74/volumes" Oct 29 23:45:18.141047 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:18.141013 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" event={"ID":"2c26d97b-29bd-4a99-813a-91cd764b3199","Type":"ContainerStarted","Data":"30f614aea1c88812bb68da9ed4d523cc9f8819dc128ddd4c105454be4cd6b7a4"} Oct 29 23:45:18.143527 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:18.143498 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" event={"ID":"5d0cbcca-d158-4dbc-88f0-e827b54c630e","Type":"ContainerStarted","Data":"a705cccb3970c58db31eacdf168e44331366a951e498e672a3e54ae0d2a2d97f"} Oct 29 23:45:18.146255 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:18.146234 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" event={"ID":"01fce6be-41f6-4874-b280-6f0422b87a06","Type":"ContainerStarted","Data":"04ad12079fc6662beb7cca2d6ca43d380065ef1d7a86f3a21a52eed1fb639177"} Oct 29 23:45:18.151374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:18.151346 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb" event={"ID":"f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142","Type":"ContainerStarted","Data":"c7938d886a0948725100dba05ac5c9de6cf5982a7be2009886e9962a187b8bcd"} Oct 29 23:45:18.156294 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:18.156272 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" event={"ID":"ca325758-625b-4f60-9b98-9a34cb9c35b7","Type":"ContainerStarted","Data":"826903184a6907f6c091c00ca783b37c548a1a940ee6d5853dd8ff2d2a7efae3"} Oct 29 23:45:18.328433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:18.328329 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb" podStartSLOduration=18.328312938 podStartE2EDuration="18.328312938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:18.317921741 +0000 UTC m=+3764.969420271" watchObservedRunningTime="2026-10-29 23:45:18.328312938 +0000 UTC m=+3764.979811469" Oct 29 23:45:18.474397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:18.474367 5057 scope.go:122] "RemoveContainer" containerID="5b644524e0bafc859fdf8b99e53ab162e6280eb4e1b3316a128eb1191fa9d545" Oct 29 23:45:18.474608 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:18.474548 5057 scope.go:122] "RemoveContainer" containerID="43a03ebf3235b729925abe2361c0f28452f94f42002ebecaf2d71311d26d1c83" Oct 29 23:45:19.170699 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.170664 5057 generic.go:356] "Generic (PLEG): container finished" podID="ca325758-625b-4f60-9b98-9a34cb9c35b7" containerID="826903184a6907f6c091c00ca783b37c548a1a940ee6d5853dd8ff2d2a7efae3" exitCode=60 Oct 29 23:45:19.171306 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.170738 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" event={"ID":"ca325758-625b-4f60-9b98-9a34cb9c35b7","Type":"ContainerDied","Data":"826903184a6907f6c091c00ca783b37c548a1a940ee6d5853dd8ff2d2a7efae3"} Oct 29 23:45:19.171306 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.170799 5057 scope.go:122] "RemoveContainer" containerID="54dc9624534f2c3e30edb85a6d99b9beae2f30cee4fba59cae98852605439b76" Oct 29 23:45:19.171306 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.171289 5057 scope.go:122] "RemoveContainer" containerID="826903184a6907f6c091c00ca783b37c548a1a940ee6d5853dd8ff2d2a7efae3" Oct 29 23:45:19.171774 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:19.171748 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888625-jpzwv_odh-ai-gateway-infra(ca325758-625b-4f60-9b98-9a34cb9c35b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" podUID="ca325758-625b-4f60-9b98-9a34cb9c35b7" Oct 29 23:45:19.176415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.176390 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" event={"ID":"1d322efb-9ed7-4868-b1b5-6ac26e0ce34d","Type":"ContainerStarted","Data":"e3ddeaa4b6af04f1f2aeefb075a49c2234e694e5401dfb3afa3da860267706b0"} Oct 29 23:45:19.178830 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.178799 5057 generic.go:356] "Generic (PLEG): container finished" podID="cca7fdd3-eac8-49b9-bb6a-a621194a686b" containerID="e50220e9d3b8c8d2aa8eb29093baf393beffae8cce00d1f7e6727d205e9ee3d1" exitCode=60 Oct 29 23:45:19.178966 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.178877 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj" event={"ID":"cca7fdd3-eac8-49b9-bb6a-a621194a686b","Type":"ContainerDied","Data":"e50220e9d3b8c8d2aa8eb29093baf393beffae8cce00d1f7e6727d205e9ee3d1"} Oct 29 23:45:19.179272 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.179254 5057 scope.go:122] "RemoveContainer" containerID="e50220e9d3b8c8d2aa8eb29093baf393beffae8cce00d1f7e6727d205e9ee3d1" Oct 29 23:45:19.179671 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:19.179647 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj_odh-ai-gateway-infra(cca7fdd3-eac8-49b9-bb6a-a621194a686b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj" podUID="cca7fdd3-eac8-49b9-bb6a-a621194a686b" Oct 29 23:45:19.181114 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.181093 5057 generic.go:356] "Generic (PLEG): container finished" podID="2c26d97b-29bd-4a99-813a-91cd764b3199" containerID="30f614aea1c88812bb68da9ed4d523cc9f8819dc128ddd4c105454be4cd6b7a4" exitCode=60 Oct 29 23:45:19.181266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.181167 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" event={"ID":"2c26d97b-29bd-4a99-813a-91cd764b3199","Type":"ContainerDied","Data":"30f614aea1c88812bb68da9ed4d523cc9f8819dc128ddd4c105454be4cd6b7a4"} Oct 29 23:45:19.181591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.181564 5057 scope.go:122] "RemoveContainer" containerID="30f614aea1c88812bb68da9ed4d523cc9f8819dc128ddd4c105454be4cd6b7a4" Oct 29 23:45:19.181943 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:19.181916 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5_odh-ai-gateway-infra(2c26d97b-29bd-4a99-813a-91cd764b3199)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" podUID="2c26d97b-29bd-4a99-813a-91cd764b3199" Oct 29 23:45:19.183381 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.183358 5057 generic.go:356] "Generic (PLEG): container finished" podID="5d0cbcca-d158-4dbc-88f0-e827b54c630e" containerID="a705cccb3970c58db31eacdf168e44331366a951e498e672a3e54ae0d2a2d97f" exitCode=60 Oct 29 23:45:19.183498 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.183434 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" event={"ID":"5d0cbcca-d158-4dbc-88f0-e827b54c630e","Type":"ContainerDied","Data":"a705cccb3970c58db31eacdf168e44331366a951e498e672a3e54ae0d2a2d97f"} Oct 29 23:45:19.183740 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.183724 5057 scope.go:122] "RemoveContainer" containerID="a705cccb3970c58db31eacdf168e44331366a951e498e672a3e54ae0d2a2d97f" Oct 29 23:45:19.184013 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:19.183984 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r_odh-ai-gateway-infra(5d0cbcca-d158-4dbc-88f0-e827b54c630e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" podUID="5d0cbcca-d158-4dbc-88f0-e827b54c630e" Oct 29 23:45:19.185833 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.185815 5057 generic.go:356] "Generic (PLEG): container finished" podID="2dc6fc01-4438-439e-8106-2a0c36a73f73" containerID="565a340234ce6de345f395dd3cf1b3747ec15602aff71230063f387915ea7d83" exitCode=60 Oct 29 23:45:19.185981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.185891 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8" event={"ID":"2dc6fc01-4438-439e-8106-2a0c36a73f73","Type":"ContainerDied","Data":"565a340234ce6de345f395dd3cf1b3747ec15602aff71230063f387915ea7d83"} Oct 29 23:45:19.186262 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.186243 5057 scope.go:122] "RemoveContainer" containerID="565a340234ce6de345f395dd3cf1b3747ec15602aff71230063f387915ea7d83" Oct 29 23:45:19.186542 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:19.186520 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8_odh-ai-gateway-infra(2dc6fc01-4438-439e-8106-2a0c36a73f73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8" podUID="2dc6fc01-4438-439e-8106-2a0c36a73f73" Oct 29 23:45:19.187905 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.187886 5057 generic.go:356] "Generic (PLEG): container finished" podID="01fce6be-41f6-4874-b280-6f0422b87a06" containerID="04ad12079fc6662beb7cca2d6ca43d380065ef1d7a86f3a21a52eed1fb639177" exitCode=60 Oct 29 23:45:19.187994 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.187965 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" event={"ID":"01fce6be-41f6-4874-b280-6f0422b87a06","Type":"ContainerDied","Data":"04ad12079fc6662beb7cca2d6ca43d380065ef1d7a86f3a21a52eed1fb639177"} Oct 29 23:45:19.188382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.188364 5057 scope.go:122] "RemoveContainer" containerID="04ad12079fc6662beb7cca2d6ca43d380065ef1d7a86f3a21a52eed1fb639177" Oct 29 23:45:19.188644 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:19.188618 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z_odh-ai-gateway-infra(01fce6be-41f6-4874-b280-6f0422b87a06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" podUID="01fce6be-41f6-4874-b280-6f0422b87a06" Oct 29 23:45:19.190079 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.190058 5057 generic.go:356] "Generic (PLEG): container finished" podID="f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142" containerID="c7938d886a0948725100dba05ac5c9de6cf5982a7be2009886e9962a187b8bcd" exitCode=60 Oct 29 23:45:19.190155 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.190095 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb" event={"ID":"f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142","Type":"ContainerDied","Data":"c7938d886a0948725100dba05ac5c9de6cf5982a7be2009886e9962a187b8bcd"} Oct 29 23:45:19.190475 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.190456 5057 scope.go:122] "RemoveContainer" containerID="c7938d886a0948725100dba05ac5c9de6cf5982a7be2009886e9962a187b8bcd" Oct 29 23:45:19.191117 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:19.191092 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb_odh-ai-gateway-infra(f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb" podUID="f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142" Oct 29 23:45:19.193106 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.193083 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" event={"ID":"e2707a1f-9602-475e-83ea-df56bcf01224","Type":"ContainerStarted","Data":"d276e8512bb5ddbff614fbf893e2af18afa53a442e395cd7b74eb848f022ce6a"} Oct 29 23:45:19.196658 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.196636 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5"] Oct 29 23:45:19.315466 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.291323 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv"] Oct 29 23:45:19.608010 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.523215 5057 scope.go:122] "RemoveContainer" containerID="e2c29de89ed17785b9460425ab2c1cf9cda959e27ed6e7756d854ee85c1b6899" Oct 29 23:45:19.608010 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.525286 5057 scope.go:122] "RemoveContainer" containerID="49b3ce504664c7cecc01749771a852db2d358f76dc52dc053d39ace690c1d383" Oct 29 23:45:19.608010 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.586567 5057 scope.go:122] "RemoveContainer" containerID="670122541b237b55e3fa938fb12a53ef98905b273dcaad28d943017308adcdf7" Oct 29 23:45:19.650211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.610755 5057 scope.go:122] "RemoveContainer" containerID="fb700a84a96f8ad6c29f52339e3499900070a062c7d977d06a429792019ad62b" Oct 29 23:45:19.782636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.782605 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z"] Oct 29 23:45:19.786567 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.786546 5057 scope.go:122] "RemoveContainer" containerID="5dbd9c21aa2b9be412d28c5c5f449ed18449495ef510ff61fd7f8d3a5676e9b0" Oct 29 23:45:19.801347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.801321 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb"] Oct 29 23:45:19.802354 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.802333 5057 scope.go:122] "RemoveContainer" containerID="eeb7c616d0f62610e21e2f0b294fd5961b6ccea55e62f727709752b9094d1e17" Oct 29 23:45:19.820124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.820096 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r"] Oct 29 23:45:19.820413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.820386 5057 scope.go:122] "RemoveContainer" containerID="b49dddb6d097010bb5518b938ba1d3224f5af4bdb82419c53824e1a6cbf69fd8" Oct 29 23:45:19.840238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.840212 5057 scope.go:122] "RemoveContainer" containerID="fed549c7efff5e53518a4aa9646d8622d5aa6050b927bb67e5648a2e38acfd18" Oct 29 23:45:19.860981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:19.860882 5057 scope.go:122] "RemoveContainer" containerID="8447e37d6aeffa922849626901ada9bbf63bcf17e22edcb5b3933446d05654f1" Oct 29 23:45:20.204088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.204056 5057 generic.go:356] "Generic (PLEG): container finished" podID="1d322efb-9ed7-4868-b1b5-6ac26e0ce34d" containerID="e3ddeaa4b6af04f1f2aeefb075a49c2234e694e5401dfb3afa3da860267706b0" exitCode=60 Oct 29 23:45:20.204594 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.204128 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" event={"ID":"1d322efb-9ed7-4868-b1b5-6ac26e0ce34d","Type":"ContainerDied","Data":"e3ddeaa4b6af04f1f2aeefb075a49c2234e694e5401dfb3afa3da860267706b0"} Oct 29 23:45:20.204594 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.204200 5057 scope.go:122] "RemoveContainer" containerID="c5b4b74a4534e573560ccde8a9f510eaa9385ffa77a1711d5111f605da604ae6" Oct 29 23:45:20.205169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.205127 5057 scope.go:122] "RemoveContainer" containerID="e3ddeaa4b6af04f1f2aeefb075a49c2234e694e5401dfb3afa3da860267706b0" Oct 29 23:45:20.205642 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:20.205618 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh_odh-ai-gateway-infra(1d322efb-9ed7-4868-b1b5-6ac26e0ce34d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" podUID="1d322efb-9ed7-4868-b1b5-6ac26e0ce34d" Oct 29 23:45:20.208175 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.208150 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" event={"ID":"848ba138-a799-4dab-b704-2ae74e04ac95","Type":"ContainerStarted","Data":"1002b58c271a7d7132fca47eddf28b6aec42181f5f93121cc0c5444274567a1a"} Oct 29 23:45:20.219936 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.219659 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" event={"ID":"4d541e01-8a85-4dd3-801f-9a44798a0ed0","Type":"ContainerStarted","Data":"664141a556ed144ea4650fd1983c12d08e4732ceb014d63db783b3f64dc759b2"} Oct 29 23:45:20.233624 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.233592 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" event={"ID":"e2707a1f-9602-475e-83ea-df56bcf01224","Type":"ContainerDied","Data":"d276e8512bb5ddbff614fbf893e2af18afa53a442e395cd7b74eb848f022ce6a"} Oct 29 23:45:20.233743 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.233663 5057 scope.go:122] "RemoveContainer" containerID="f9867c09d2c81255d094ba5162971ca7fb24f4ec52a16c7c5a35dde39fd7801a" Oct 29 23:45:20.234174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.234156 5057 scope.go:122] "RemoveContainer" containerID="d276e8512bb5ddbff614fbf893e2af18afa53a442e395cd7b74eb848f022ce6a" Oct 29 23:45:20.234785 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:20.234760 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5_odh-ai-gateway-infra(e2707a1f-9602-475e-83ea-df56bcf01224)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" podUID="e2707a1f-9602-475e-83ea-df56bcf01224" Oct 29 23:45:20.235317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.233546 5057 generic.go:356] "Generic (PLEG): container finished" podID="e2707a1f-9602-475e-83ea-df56bcf01224" containerID="d276e8512bb5ddbff614fbf893e2af18afa53a442e395cd7b74eb848f022ce6a" exitCode=60 Oct 29 23:45:20.295521 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.295443 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5"] Oct 29 23:45:20.604109 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.604075 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" Oct 29 23:45:20.611042 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.610957 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" Oct 29 23:45:20.629459 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.629371 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" Oct 29 23:45:20.658145 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.658110 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh"] Oct 29 23:45:20.684642 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.684611 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c26d97b-29bd-4a99-813a-91cd764b3199-kube-api-access-58tkn\" (UniqueName: \"kubernetes.io/projected/2c26d97b-29bd-4a99-813a-91cd764b3199-kube-api-access-58tkn\") pod \"2c26d97b-29bd-4a99-813a-91cd764b3199\" (UID: \"2c26d97b-29bd-4a99-813a-91cd764b3199\") " Oct 29 23:45:20.684805 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.684658 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01fce6be-41f6-4874-b280-6f0422b87a06-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fce6be-41f6-4874-b280-6f0422b87a06-maas-api-service-ca\") pod \"01fce6be-41f6-4874-b280-6f0422b87a06\" (UID: \"01fce6be-41f6-4874-b280-6f0422b87a06\") " Oct 29 23:45:20.684805 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.684727 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0cbcca-d158-4dbc-88f0-e827b54c630e-kube-api-access-g7k8g\" (UniqueName: \"kubernetes.io/projected/5d0cbcca-d158-4dbc-88f0-e827b54c630e-kube-api-access-g7k8g\") pod \"5d0cbcca-d158-4dbc-88f0-e827b54c630e\" (UID: \"5d0cbcca-d158-4dbc-88f0-e827b54c630e\") " Oct 29 23:45:20.684805 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.684757 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c26d97b-29bd-4a99-813a-91cd764b3199-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c26d97b-29bd-4a99-813a-91cd764b3199-maas-api-service-ca\") pod \"2c26d97b-29bd-4a99-813a-91cd764b3199\" (UID: \"2c26d97b-29bd-4a99-813a-91cd764b3199\") " Oct 29 23:45:20.684946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.684935 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d0cbcca-d158-4dbc-88f0-e827b54c630e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d0cbcca-d158-4dbc-88f0-e827b54c630e-maas-api-service-ca\") pod \"5d0cbcca-d158-4dbc-88f0-e827b54c630e\" (UID: \"5d0cbcca-d158-4dbc-88f0-e827b54c630e\") " Oct 29 23:45:20.684981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.684958 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01fce6be-41f6-4874-b280-6f0422b87a06-kube-api-access-kdj94\" (UniqueName: \"kubernetes.io/projected/01fce6be-41f6-4874-b280-6f0422b87a06-kube-api-access-kdj94\") pod \"01fce6be-41f6-4874-b280-6f0422b87a06\" (UID: \"01fce6be-41f6-4874-b280-6f0422b87a06\") " Oct 29 23:45:20.686673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.686629 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0cbcca-d158-4dbc-88f0-e827b54c630e-maas-api-service-ca" pod "5d0cbcca-d158-4dbc-88f0-e827b54c630e" (UID: "5d0cbcca-d158-4dbc-88f0-e827b54c630e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:20.687176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.687155 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01fce6be-41f6-4874-b280-6f0422b87a06-maas-api-service-ca" pod "01fce6be-41f6-4874-b280-6f0422b87a06" (UID: "01fce6be-41f6-4874-b280-6f0422b87a06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:20.687669 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.687651 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c26d97b-29bd-4a99-813a-91cd764b3199-maas-api-service-ca" pod "2c26d97b-29bd-4a99-813a-91cd764b3199" (UID: "2c26d97b-29bd-4a99-813a-91cd764b3199"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:20.688905 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.688877 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fce6be-41f6-4874-b280-6f0422b87a06-kube-api-access-kdj94" pod "01fce6be-41f6-4874-b280-6f0422b87a06" (UID: "01fce6be-41f6-4874-b280-6f0422b87a06"). InnerVolumeSpecName "kube-api-access-kdj94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:20.689338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.689314 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c26d97b-29bd-4a99-813a-91cd764b3199-kube-api-access-58tkn" pod "2c26d97b-29bd-4a99-813a-91cd764b3199" (UID: "2c26d97b-29bd-4a99-813a-91cd764b3199"). InnerVolumeSpecName "kube-api-access-58tkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:20.690102 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.690087 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0cbcca-d158-4dbc-88f0-e827b54c630e-kube-api-access-g7k8g" pod "5d0cbcca-d158-4dbc-88f0-e827b54c630e" (UID: "5d0cbcca-d158-4dbc-88f0-e827b54c630e"). InnerVolumeSpecName "kube-api-access-g7k8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:20.690878 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.690793 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb" Oct 29 23:45:20.699217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.699195 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" Oct 29 23:45:20.780005 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.779965 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8"] Oct 29 23:45:20.786612 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.786588 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca325758-625b-4f60-9b98-9a34cb9c35b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca325758-625b-4f60-9b98-9a34cb9c35b7-maas-api-service-ca\") pod \"ca325758-625b-4f60-9b98-9a34cb9c35b7\" (UID: \"ca325758-625b-4f60-9b98-9a34cb9c35b7\") " Oct 29 23:45:20.786897 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.786870 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142-maas-api-service-ca\") pod \"f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142\" (UID: \"f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142\") " Oct 29 23:45:20.787749 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.787434 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142-maas-api-service-ca" pod "f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142" (UID: "f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:20.787864 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.787649 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca325758-625b-4f60-9b98-9a34cb9c35b7-maas-api-service-ca" pod "ca325758-625b-4f60-9b98-9a34cb9c35b7" (UID: "ca325758-625b-4f60-9b98-9a34cb9c35b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:20.788021 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.788000 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142-kube-api-access-dkjqc\" (UniqueName: \"kubernetes.io/projected/f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142-kube-api-access-dkjqc\") pod \"f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142\" (UID: \"f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142\") " Oct 29 23:45:20.788653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.788608 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca325758-625b-4f60-9b98-9a34cb9c35b7-kube-api-access-56gdd\" (UniqueName: \"kubernetes.io/projected/ca325758-625b-4f60-9b98-9a34cb9c35b7-kube-api-access-56gdd\") pod \"ca325758-625b-4f60-9b98-9a34cb9c35b7\" (UID: \"ca325758-625b-4f60-9b98-9a34cb9c35b7\") " Oct 29 23:45:20.790485 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.790467 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d0cbcca-d158-4dbc-88f0-e827b54c630e-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.790612 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.790600 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdj94\" (UniqueName: \"kubernetes.io/projected/01fce6be-41f6-4874-b280-6f0422b87a06-kube-api-access-kdj94\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.790682 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.790672 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58tkn\" (UniqueName: \"kubernetes.io/projected/2c26d97b-29bd-4a99-813a-91cd764b3199-kube-api-access-58tkn\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.790748 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.790738 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fce6be-41f6-4874-b280-6f0422b87a06-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.790808 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.790800 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7k8g\" (UniqueName: \"kubernetes.io/projected/5d0cbcca-d158-4dbc-88f0-e827b54c630e-kube-api-access-g7k8g\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.790869 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.790861 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c26d97b-29bd-4a99-813a-91cd764b3199-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.790934 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.790926 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca325758-625b-4f60-9b98-9a34cb9c35b7-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.790993 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.790983 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.794939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.794913 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca325758-625b-4f60-9b98-9a34cb9c35b7-kube-api-access-56gdd" pod "ca325758-625b-4f60-9b98-9a34cb9c35b7" (UID: "ca325758-625b-4f60-9b98-9a34cb9c35b7"). InnerVolumeSpecName "kube-api-access-56gdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:20.796363 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.796342 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142-kube-api-access-dkjqc" pod "f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142" (UID: "f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142"). InnerVolumeSpecName "kube-api-access-dkjqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:20.843277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.842656 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj"] Oct 29 23:45:20.894964 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.894890 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkjqc\" (UniqueName: \"kubernetes.io/projected/f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142-kube-api-access-dkjqc\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.894964 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:20.894927 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56gdd\" (UniqueName: \"kubernetes.io/projected/ca325758-625b-4f60-9b98-9a34cb9c35b7-kube-api-access-56gdd\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:21.212941 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.212907 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8" Oct 29 23:45:21.217896 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.217871 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj" Oct 29 23:45:21.284346 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.284235 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8" event={"ID":"2dc6fc01-4438-439e-8106-2a0c36a73f73","Type":"ContainerDied","Data":"83dc77f2c45cbb75b6ad3ae0fa57c3d0613f6c51269a5082db5b2a55f77507d1"} Oct 29 23:45:21.284591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.284577 5057 scope.go:122] "RemoveContainer" containerID="565a340234ce6de345f395dd3cf1b3747ec15602aff71230063f387915ea7d83" Oct 29 23:45:21.284733 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.284251 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8" Oct 29 23:45:21.290096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.290067 5057 generic.go:356] "Generic (PLEG): container finished" podID="da38ccbd-b489-4225-a4d1-4502943eb3d4" containerID="f360ea40c904cc8df338e27c38f481a55ec35ddfaffd66ce7557560a48650221" exitCode=60 Oct 29 23:45:21.290366 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.290341 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" event={"ID":"da38ccbd-b489-4225-a4d1-4502943eb3d4","Type":"ContainerDied","Data":"f360ea40c904cc8df338e27c38f481a55ec35ddfaffd66ce7557560a48650221"} Oct 29 23:45:21.291037 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.291016 5057 scope.go:122] "RemoveContainer" containerID="f360ea40c904cc8df338e27c38f481a55ec35ddfaffd66ce7557560a48650221" Oct 29 23:45:21.294795 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.294764 5057 generic.go:356] "Generic (PLEG): container finished" podID="4d541e01-8a85-4dd3-801f-9a44798a0ed0" containerID="664141a556ed144ea4650fd1983c12d08e4732ceb014d63db783b3f64dc759b2" exitCode=60 Oct 29 23:45:21.294945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.294887 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" event={"ID":"4d541e01-8a85-4dd3-801f-9a44798a0ed0","Type":"ContainerDied","Data":"664141a556ed144ea4650fd1983c12d08e4732ceb014d63db783b3f64dc759b2"} Oct 29 23:45:21.295358 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:21.295332 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz_odh-ai-gateway-infra(da38ccbd-b489-4225-a4d1-4502943eb3d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" podUID="da38ccbd-b489-4225-a4d1-4502943eb3d4" Oct 29 23:45:21.295443 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.295380 5057 scope.go:122] "RemoveContainer" containerID="664141a556ed144ea4650fd1983c12d08e4732ceb014d63db783b3f64dc759b2" Oct 29 23:45:21.295903 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:21.295876 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b_odh-ai-gateway-infra(4d541e01-8a85-4dd3-801f-9a44798a0ed0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" podUID="4d541e01-8a85-4dd3-801f-9a44798a0ed0" Oct 29 23:45:21.303393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.303355 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dc6fc01-4438-439e-8106-2a0c36a73f73-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dc6fc01-4438-439e-8106-2a0c36a73f73-maas-api-service-ca\") pod \"2dc6fc01-4438-439e-8106-2a0c36a73f73\" (UID: \"2dc6fc01-4438-439e-8106-2a0c36a73f73\") " Oct 29 23:45:21.303467 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.303404 5057 scope.go:122] "RemoveContainer" containerID="49b3ce504664c7cecc01749771a852db2d358f76dc52dc053d39ace690c1d383" Oct 29 23:45:21.303467 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.303412 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca7fdd3-eac8-49b9-bb6a-a621194a686b-kube-api-access-sjbq4\" (UniqueName: \"kubernetes.io/projected/cca7fdd3-eac8-49b9-bb6a-a621194a686b-kube-api-access-sjbq4\") pod \"cca7fdd3-eac8-49b9-bb6a-a621194a686b\" (UID: \"cca7fdd3-eac8-49b9-bb6a-a621194a686b\") " Oct 29 23:45:21.303569 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.303548 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc6fc01-4438-439e-8106-2a0c36a73f73-kube-api-access-gct45\" (UniqueName: \"kubernetes.io/projected/2dc6fc01-4438-439e-8106-2a0c36a73f73-kube-api-access-gct45\") pod \"2dc6fc01-4438-439e-8106-2a0c36a73f73\" (UID: \"2dc6fc01-4438-439e-8106-2a0c36a73f73\") " Oct 29 23:45:21.303626 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.303598 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cca7fdd3-eac8-49b9-bb6a-a621194a686b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca7fdd3-eac8-49b9-bb6a-a621194a686b-maas-api-service-ca\") pod \"cca7fdd3-eac8-49b9-bb6a-a621194a686b\" (UID: \"cca7fdd3-eac8-49b9-bb6a-a621194a686b\") " Oct 29 23:45:21.303802 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.303775 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc6fc01-4438-439e-8106-2a0c36a73f73-maas-api-service-ca" pod "2dc6fc01-4438-439e-8106-2a0c36a73f73" (UID: "2dc6fc01-4438-439e-8106-2a0c36a73f73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:21.304417 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.304258 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca7fdd3-eac8-49b9-bb6a-a621194a686b-maas-api-service-ca" pod "cca7fdd3-eac8-49b9-bb6a-a621194a686b" (UID: "cca7fdd3-eac8-49b9-bb6a-a621194a686b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:21.304688 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.304636 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca7fdd3-eac8-49b9-bb6a-a621194a686b-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:21.304688 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.304659 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dc6fc01-4438-439e-8106-2a0c36a73f73-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:21.305719 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.305695 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" Oct 29 23:45:21.306268 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.306172 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z" event={"ID":"01fce6be-41f6-4874-b280-6f0422b87a06","Type":"ContainerDied","Data":"0d5c7a0499d6b6edc5301d1da5b87abf971c99872a0c9721abc93f4a06b4c920"} Oct 29 23:45:21.309517 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.309168 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb" event={"ID":"f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142","Type":"ContainerDied","Data":"094dba3686b0d2062de934a580ffb606f79b6403965af0b9c006b5b29f0ab04f"} Oct 29 23:45:21.309517 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.309179 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb" Oct 29 23:45:21.310678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.310636 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca7fdd3-eac8-49b9-bb6a-a621194a686b-kube-api-access-sjbq4" pod "cca7fdd3-eac8-49b9-bb6a-a621194a686b" (UID: "cca7fdd3-eac8-49b9-bb6a-a621194a686b"). InnerVolumeSpecName "kube-api-access-sjbq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:21.313019 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.312994 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc6fc01-4438-439e-8106-2a0c36a73f73-kube-api-access-gct45" pod "2dc6fc01-4438-439e-8106-2a0c36a73f73" (UID: "2dc6fc01-4438-439e-8106-2a0c36a73f73"). InnerVolumeSpecName "kube-api-access-gct45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:21.318694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.317772 5057 generic.go:356] "Generic (PLEG): container finished" podID="759dfe21-e08a-42ae-a63d-097ada97aa15" containerID="8bda2e981f761905adeefcc1eca3d785ea88b07eb3804fd604415a071ea0cbfd" exitCode=60 Oct 29 23:45:21.318694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.317877 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" event={"ID":"759dfe21-e08a-42ae-a63d-097ada97aa15","Type":"ContainerDied","Data":"8bda2e981f761905adeefcc1eca3d785ea88b07eb3804fd604415a071ea0cbfd"} Oct 29 23:45:21.318694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.318407 5057 scope.go:122] "RemoveContainer" containerID="8bda2e981f761905adeefcc1eca3d785ea88b07eb3804fd604415a071ea0cbfd" Oct 29 23:45:21.319712 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:21.319616 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5_odh-ai-gateway-infra(759dfe21-e08a-42ae-a63d-097ada97aa15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" podUID="759dfe21-e08a-42ae-a63d-097ada97aa15" Oct 29 23:45:21.321568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.321542 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" event={"ID":"ca325758-625b-4f60-9b98-9a34cb9c35b7","Type":"ContainerDied","Data":"03614fe698382afcdd516f509a0081126a0f2b51eb429818c0ebfb92fef81be8"} Oct 29 23:45:21.321687 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.321660 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv" Oct 29 23:45:21.339586 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.339332 5057 scope.go:122] "RemoveContainer" containerID="43a03ebf3235b729925abe2361c0f28452f94f42002ebecaf2d71311d26d1c83" Oct 29 23:45:21.340248 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.340227 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj" Oct 29 23:45:21.340511 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.340268 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj" event={"ID":"cca7fdd3-eac8-49b9-bb6a-a621194a686b","Type":"ContainerDied","Data":"138f98c167aa2763bedc8db1c2a0bdc1f1cc8c295c909240eeac891133369588"} Oct 29 23:45:21.343599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.343567 5057 generic.go:356] "Generic (PLEG): container finished" podID="848ba138-a799-4dab-b704-2ae74e04ac95" containerID="1002b58c271a7d7132fca47eddf28b6aec42181f5f93121cc0c5444274567a1a" exitCode=60 Oct 29 23:45:21.343716 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.343667 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" event={"ID":"848ba138-a799-4dab-b704-2ae74e04ac95","Type":"ContainerDied","Data":"1002b58c271a7d7132fca47eddf28b6aec42181f5f93121cc0c5444274567a1a"} Oct 29 23:45:21.344174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.344153 5057 scope.go:122] "RemoveContainer" containerID="1002b58c271a7d7132fca47eddf28b6aec42181f5f93121cc0c5444274567a1a" Oct 29 23:45:21.344716 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:21.344689 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5_odh-ai-gateway-infra(848ba138-a799-4dab-b704-2ae74e04ac95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" podUID="848ba138-a799-4dab-b704-2ae74e04ac95" Oct 29 23:45:21.350461 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.350376 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" event={"ID":"2c26d97b-29bd-4a99-813a-91cd764b3199","Type":"ContainerDied","Data":"3a13f84792c6113f8e88f95153b5024b0fee3f3c8ebfa58fdf2d2cd93ca640fb"} Oct 29 23:45:21.350551 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.350477 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5" Oct 29 23:45:21.356234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.355670 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b"] Oct 29 23:45:21.363348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.363313 5057 generic.go:356] "Generic (PLEG): container finished" podID="2b0ed631-9976-4f08-8d68-e783428c6a48" containerID="930ec3be00e3078706d3917d8e7778862a2bc5412a4ef8d3c55e72faef56ac34" exitCode=60 Oct 29 23:45:21.363435 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.363413 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc" event={"ID":"2b0ed631-9976-4f08-8d68-e783428c6a48","Type":"ContainerDied","Data":"930ec3be00e3078706d3917d8e7778862a2bc5412a4ef8d3c55e72faef56ac34"} Oct 29 23:45:21.363956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.363934 5057 scope.go:122] "RemoveContainer" containerID="930ec3be00e3078706d3917d8e7778862a2bc5412a4ef8d3c55e72faef56ac34" Oct 29 23:45:21.366099 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:21.364464 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc_odh-ai-gateway-infra(2b0ed631-9976-4f08-8d68-e783428c6a48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc" podUID="2b0ed631-9976-4f08-8d68-e783428c6a48" Oct 29 23:45:21.374669 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.374646 5057 scope.go:122] "RemoveContainer" containerID="04ad12079fc6662beb7cca2d6ca43d380065ef1d7a86f3a21a52eed1fb639177" Oct 29 23:45:21.381706 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.381676 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" event={"ID":"5d0cbcca-d158-4dbc-88f0-e827b54c630e","Type":"ContainerDied","Data":"b591fc659eee8738a3d7789c81ea3b1064415aae4974e1d58fb173c663cf0311"} Oct 29 23:45:21.382745 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.382070 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r" Oct 29 23:45:21.406724 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.406682 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjbq4\" (UniqueName: \"kubernetes.io/projected/cca7fdd3-eac8-49b9-bb6a-a621194a686b-kube-api-access-sjbq4\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:21.406904 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.406889 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gct45\" (UniqueName: \"kubernetes.io/projected/2dc6fc01-4438-439e-8106-2a0c36a73f73-kube-api-access-gct45\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:21.411722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.411698 5057 scope.go:122] "RemoveContainer" containerID="c7938d886a0948725100dba05ac5c9de6cf5982a7be2009886e9962a187b8bcd" Oct 29 23:45:21.430420 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.429562 5057 scope.go:122] "RemoveContainer" containerID="e2c29de89ed17785b9460425ab2c1cf9cda959e27ed6e7756d854ee85c1b6899" Oct 29 23:45:21.458093 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.458056 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5"] Oct 29 23:45:21.491009 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.490980 5057 scope.go:122] "RemoveContainer" containerID="826903184a6907f6c091c00ca783b37c548a1a940ee6d5853dd8ff2d2a7efae3" Oct 29 23:45:21.509581 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.509555 5057 scope.go:122] "RemoveContainer" containerID="e50220e9d3b8c8d2aa8eb29093baf393beffae8cce00d1f7e6727d205e9ee3d1" Oct 29 23:45:21.554236 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.554215 5057 scope.go:122] "RemoveContainer" containerID="5b644524e0bafc859fdf8b99e53ab162e6280eb4e1b3316a128eb1191fa9d545" Oct 29 23:45:21.580705 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.580680 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" Oct 29 23:45:21.586899 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.586875 5057 scope.go:122] "RemoveContainer" containerID="30f614aea1c88812bb68da9ed4d523cc9f8819dc128ddd4c105454be4cd6b7a4" Oct 29 23:45:21.587518 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.587492 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv"] Oct 29 23:45:21.597456 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.597429 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888625-jpzwv"] Oct 29 23:45:21.606602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.606580 5057 scope.go:122] "RemoveContainer" containerID="fb700a84a96f8ad6c29f52339e3499900070a062c7d977d06a429792019ad62b" Oct 29 23:45:21.620320 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.620288 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" Oct 29 23:45:21.626727 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.626702 5057 scope.go:122] "RemoveContainer" containerID="a705cccb3970c58db31eacdf168e44331366a951e498e672a3e54ae0d2a2d97f" Oct 29 23:45:21.674918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.674886 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z"] Oct 29 23:45:21.691970 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.691929 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888625-jzp5z"] Oct 29 23:45:21.711358 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.711322 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2707a1f-9602-475e-83ea-df56bcf01224-kube-api-access-5zmgd\" (UniqueName: \"kubernetes.io/projected/e2707a1f-9602-475e-83ea-df56bcf01224-kube-api-access-5zmgd\") pod \"e2707a1f-9602-475e-83ea-df56bcf01224\" (UID: \"e2707a1f-9602-475e-83ea-df56bcf01224\") " Oct 29 23:45:21.711572 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.711469 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2707a1f-9602-475e-83ea-df56bcf01224-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2707a1f-9602-475e-83ea-df56bcf01224-maas-api-service-ca\") pod \"e2707a1f-9602-475e-83ea-df56bcf01224\" (UID: \"e2707a1f-9602-475e-83ea-df56bcf01224\") " Oct 29 23:45:21.712448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.711743 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d322efb-9ed7-4868-b1b5-6ac26e0ce34d-kube-api-access-j2hhr\" (UniqueName: \"kubernetes.io/projected/1d322efb-9ed7-4868-b1b5-6ac26e0ce34d-kube-api-access-j2hhr\") pod \"1d322efb-9ed7-4868-b1b5-6ac26e0ce34d\" (UID: \"1d322efb-9ed7-4868-b1b5-6ac26e0ce34d\") " Oct 29 23:45:21.712448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.711904 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2707a1f-9602-475e-83ea-df56bcf01224-maas-api-service-ca" pod "e2707a1f-9602-475e-83ea-df56bcf01224" (UID: "e2707a1f-9602-475e-83ea-df56bcf01224"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:21.712448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.711933 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d322efb-9ed7-4868-b1b5-6ac26e0ce34d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d322efb-9ed7-4868-b1b5-6ac26e0ce34d-maas-api-service-ca\") pod \"1d322efb-9ed7-4868-b1b5-6ac26e0ce34d\" (UID: \"1d322efb-9ed7-4868-b1b5-6ac26e0ce34d\") " Oct 29 23:45:21.712895 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.712753 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d322efb-9ed7-4868-b1b5-6ac26e0ce34d-maas-api-service-ca" pod "1d322efb-9ed7-4868-b1b5-6ac26e0ce34d" (UID: "1d322efb-9ed7-4868-b1b5-6ac26e0ce34d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:21.713040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.712947 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d322efb-9ed7-4868-b1b5-6ac26e0ce34d-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:21.713040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.712967 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2707a1f-9602-475e-83ea-df56bcf01224-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:21.714393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.714365 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2707a1f-9602-475e-83ea-df56bcf01224-kube-api-access-5zmgd" pod "e2707a1f-9602-475e-83ea-df56bcf01224" (UID: "e2707a1f-9602-475e-83ea-df56bcf01224"). InnerVolumeSpecName "kube-api-access-5zmgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:21.715423 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.715398 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d322efb-9ed7-4868-b1b5-6ac26e0ce34d-kube-api-access-j2hhr" pod "1d322efb-9ed7-4868-b1b5-6ac26e0ce34d" (UID: "1d322efb-9ed7-4868-b1b5-6ac26e0ce34d"). InnerVolumeSpecName "kube-api-access-j2hhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:21.736848 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.736812 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5"] Oct 29 23:45:21.745501 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.745469 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888625-97zk5"] Oct 29 23:45:21.814525 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.814445 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zmgd\" (UniqueName: \"kubernetes.io/projected/e2707a1f-9602-475e-83ea-df56bcf01224-kube-api-access-5zmgd\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:21.814525 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.814473 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2hhr\" (UniqueName: \"kubernetes.io/projected/1d322efb-9ed7-4868-b1b5-6ac26e0ce34d-kube-api-access-j2hhr\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:21.827427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.827390 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb"] Oct 29 23:45:21.834903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.834872 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-26a2a4-29888625-rjhnb"] Oct 29 23:45:21.882236 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.882203 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj"] Oct 29 23:45:21.887113 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.887085 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888625-cwvwj"] Oct 29 23:45:21.923109 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.923056 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r"] Oct 29 23:45:21.931212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:21.931166 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888625-nkw5r"] Oct 29 23:45:22.114829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.114733 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8"] Oct 29 23:45:22.120809 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.120779 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-825472-29888625-gz6x8"] Oct 29 23:45:22.382731 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.382648 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz"] Oct 29 23:45:22.415477 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.413738 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" event={"ID":"e2707a1f-9602-475e-83ea-df56bcf01224","Type":"ContainerDied","Data":"9f8781f123aaf72e9078fe8761e09dfcf2966394bbcf4c5b15c151492a9c1a51"} Oct 29 23:45:22.415477 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.413803 5057 scope.go:122] "RemoveContainer" containerID="d276e8512bb5ddbff614fbf893e2af18afa53a442e395cd7b74eb848f022ce6a" Oct 29 23:45:22.415477 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.413908 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5" Oct 29 23:45:22.432211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.431639 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" event={"ID":"1d322efb-9ed7-4868-b1b5-6ac26e0ce34d","Type":"ContainerDied","Data":"d4979cb52d06f7ae50d2f4072e205c41cf12cfd6403522e039c729ab26b758b9"} Oct 29 23:45:22.432211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.431775 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh" Oct 29 23:45:22.461209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.461161 5057 scope.go:122] "RemoveContainer" containerID="e3ddeaa4b6af04f1f2aeefb075a49c2234e694e5401dfb3afa3da860267706b0" Oct 29 23:45:22.479623 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.479590 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc"] Oct 29 23:45:22.518174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.512228 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5"] Oct 29 23:45:22.522771 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.522740 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888625-prgh5"] Oct 29 23:45:22.550892 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.550359 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5"] Oct 29 23:45:22.631906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.631844 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh"] Oct 29 23:45:22.637728 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.637646 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d4110-29888625-d7dmh"] Oct 29 23:45:22.709402 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.709371 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" Oct 29 23:45:22.777874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.777840 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d541e01-8a85-4dd3-801f-9a44798a0ed0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d541e01-8a85-4dd3-801f-9a44798a0ed0-maas-api-service-ca\") pod \"4d541e01-8a85-4dd3-801f-9a44798a0ed0\" (UID: \"4d541e01-8a85-4dd3-801f-9a44798a0ed0\") " Oct 29 23:45:22.778030 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.777893 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d541e01-8a85-4dd3-801f-9a44798a0ed0-kube-api-access-tv52m\" (UniqueName: \"kubernetes.io/projected/4d541e01-8a85-4dd3-801f-9a44798a0ed0-kube-api-access-tv52m\") pod \"4d541e01-8a85-4dd3-801f-9a44798a0ed0\" (UID: \"4d541e01-8a85-4dd3-801f-9a44798a0ed0\") " Oct 29 23:45:22.780664 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.780635 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d541e01-8a85-4dd3-801f-9a44798a0ed0-maas-api-service-ca" pod "4d541e01-8a85-4dd3-801f-9a44798a0ed0" (UID: "4d541e01-8a85-4dd3-801f-9a44798a0ed0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:22.784398 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.784372 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d541e01-8a85-4dd3-801f-9a44798a0ed0-kube-api-access-tv52m" pod "4d541e01-8a85-4dd3-801f-9a44798a0ed0" (UID: "4d541e01-8a85-4dd3-801f-9a44798a0ed0"). InnerVolumeSpecName "kube-api-access-tv52m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:22.882010 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.881981 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d541e01-8a85-4dd3-801f-9a44798a0ed0-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.882108 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.882014 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv52m\" (UniqueName: \"kubernetes.io/projected/4d541e01-8a85-4dd3-801f-9a44798a0ed0-kube-api-access-tv52m\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.955726 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.955689 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc" Oct 29 23:45:22.958298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.958276 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" Oct 29 23:45:22.969332 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.969304 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" Oct 29 23:45:22.974766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.974739 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" Oct 29 23:45:22.983625 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.983597 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b0ed631-9976-4f08-8d68-e783428c6a48-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b0ed631-9976-4f08-8d68-e783428c6a48-maas-api-service-ca\") pod \"2b0ed631-9976-4f08-8d68-e783428c6a48\" (UID: \"2b0ed631-9976-4f08-8d68-e783428c6a48\") " Oct 29 23:45:22.983755 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.983671 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b0ed631-9976-4f08-8d68-e783428c6a48-kube-api-access-tmh49\" (UniqueName: \"kubernetes.io/projected/2b0ed631-9976-4f08-8d68-e783428c6a48-kube-api-access-tmh49\") pod \"2b0ed631-9976-4f08-8d68-e783428c6a48\" (UID: \"2b0ed631-9976-4f08-8d68-e783428c6a48\") " Oct 29 23:45:22.983837 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.983819 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/848ba138-a799-4dab-b704-2ae74e04ac95-kube-api-access-vpdgw\" (UniqueName: \"kubernetes.io/projected/848ba138-a799-4dab-b704-2ae74e04ac95-kube-api-access-vpdgw\") pod \"848ba138-a799-4dab-b704-2ae74e04ac95\" (UID: \"848ba138-a799-4dab-b704-2ae74e04ac95\") " Oct 29 23:45:22.983914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.983898 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da38ccbd-b489-4225-a4d1-4502943eb3d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da38ccbd-b489-4225-a4d1-4502943eb3d4-maas-api-service-ca\") pod \"da38ccbd-b489-4225-a4d1-4502943eb3d4\" (UID: \"da38ccbd-b489-4225-a4d1-4502943eb3d4\") " Oct 29 23:45:22.984030 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.984013 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/848ba138-a799-4dab-b704-2ae74e04ac95-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/848ba138-a799-4dab-b704-2ae74e04ac95-maas-api-service-ca\") pod \"848ba138-a799-4dab-b704-2ae74e04ac95\" (UID: \"848ba138-a799-4dab-b704-2ae74e04ac95\") " Oct 29 23:45:22.984104 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.984070 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da38ccbd-b489-4225-a4d1-4502943eb3d4-kube-api-access-kkkkc\" (UniqueName: \"kubernetes.io/projected/da38ccbd-b489-4225-a4d1-4502943eb3d4-kube-api-access-kkkkc\") pod \"da38ccbd-b489-4225-a4d1-4502943eb3d4\" (UID: \"da38ccbd-b489-4225-a4d1-4502943eb3d4\") " Oct 29 23:45:22.984862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.984837 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da38ccbd-b489-4225-a4d1-4502943eb3d4-maas-api-service-ca" pod "da38ccbd-b489-4225-a4d1-4502943eb3d4" (UID: "da38ccbd-b489-4225-a4d1-4502943eb3d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:22.984993 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.984867 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/848ba138-a799-4dab-b704-2ae74e04ac95-maas-api-service-ca" pod "848ba138-a799-4dab-b704-2ae74e04ac95" (UID: "848ba138-a799-4dab-b704-2ae74e04ac95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:22.985244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.985221 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0ed631-9976-4f08-8d68-e783428c6a48-maas-api-service-ca" pod "2b0ed631-9976-4f08-8d68-e783428c6a48" (UID: "2b0ed631-9976-4f08-8d68-e783428c6a48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:22.985796 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.985773 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b0ed631-9976-4f08-8d68-e783428c6a48-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.985890 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.985800 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da38ccbd-b489-4225-a4d1-4502943eb3d4-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.985890 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.985815 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/848ba138-a799-4dab-b704-2ae74e04ac95-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.987434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.987411 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/848ba138-a799-4dab-b704-2ae74e04ac95-kube-api-access-vpdgw" pod "848ba138-a799-4dab-b704-2ae74e04ac95" (UID: "848ba138-a799-4dab-b704-2ae74e04ac95"). InnerVolumeSpecName "kube-api-access-vpdgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:22.987518 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.987488 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da38ccbd-b489-4225-a4d1-4502943eb3d4-kube-api-access-kkkkc" pod "da38ccbd-b489-4225-a4d1-4502943eb3d4" (UID: "da38ccbd-b489-4225-a4d1-4502943eb3d4"). InnerVolumeSpecName "kube-api-access-kkkkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:22.989384 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:22.989358 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b0ed631-9976-4f08-8d68-e783428c6a48-kube-api-access-tmh49" pod "2b0ed631-9976-4f08-8d68-e783428c6a48" (UID: "2b0ed631-9976-4f08-8d68-e783428c6a48"). InnerVolumeSpecName "kube-api-access-tmh49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:23.087328 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.087294 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/759dfe21-e08a-42ae-a63d-097ada97aa15-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/759dfe21-e08a-42ae-a63d-097ada97aa15-maas-api-service-ca\") pod \"759dfe21-e08a-42ae-a63d-097ada97aa15\" (UID: \"759dfe21-e08a-42ae-a63d-097ada97aa15\") " Oct 29 23:45:23.087559 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.087540 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/759dfe21-e08a-42ae-a63d-097ada97aa15-kube-api-access-zj9gh\" (UniqueName: \"kubernetes.io/projected/759dfe21-e08a-42ae-a63d-097ada97aa15-kube-api-access-zj9gh\") pod \"759dfe21-e08a-42ae-a63d-097ada97aa15\" (UID: \"759dfe21-e08a-42ae-a63d-097ada97aa15\") " Oct 29 23:45:23.088212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.088128 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmh49\" (UniqueName: \"kubernetes.io/projected/2b0ed631-9976-4f08-8d68-e783428c6a48-kube-api-access-tmh49\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:23.088212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.088147 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpdgw\" (UniqueName: \"kubernetes.io/projected/848ba138-a799-4dab-b704-2ae74e04ac95-kube-api-access-vpdgw\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:23.088212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.088157 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkkkc\" (UniqueName: \"kubernetes.io/projected/da38ccbd-b489-4225-a4d1-4502943eb3d4-kube-api-access-kkkkc\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:23.088624 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.088592 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/759dfe21-e08a-42ae-a63d-097ada97aa15-maas-api-service-ca" pod "759dfe21-e08a-42ae-a63d-097ada97aa15" (UID: "759dfe21-e08a-42ae-a63d-097ada97aa15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:23.090469 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.090438 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759dfe21-e08a-42ae-a63d-097ada97aa15-kube-api-access-zj9gh" pod "759dfe21-e08a-42ae-a63d-097ada97aa15" (UID: "759dfe21-e08a-42ae-a63d-097ada97aa15"). InnerVolumeSpecName "kube-api-access-zj9gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:23.190556 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.190514 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj9gh\" (UniqueName: \"kubernetes.io/projected/759dfe21-e08a-42ae-a63d-097ada97aa15-kube-api-access-zj9gh\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:23.190556 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.190555 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/759dfe21-e08a-42ae-a63d-097ada97aa15-maas-api-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:23.486420 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.486336 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01fce6be-41f6-4874-b280-6f0422b87a06" path="/var/lib/kubelet/pods/01fce6be-41f6-4874-b280-6f0422b87a06/volumes" Oct 29 23:45:23.487011 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.486986 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d322efb-9ed7-4868-b1b5-6ac26e0ce34d" path="/var/lib/kubelet/pods/1d322efb-9ed7-4868-b1b5-6ac26e0ce34d/volumes" Oct 29 23:45:23.487477 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.487459 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c26d97b-29bd-4a99-813a-91cd764b3199" path="/var/lib/kubelet/pods/2c26d97b-29bd-4a99-813a-91cd764b3199/volumes" Oct 29 23:45:23.488168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.488151 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc6fc01-4438-439e-8106-2a0c36a73f73" path="/var/lib/kubelet/pods/2dc6fc01-4438-439e-8106-2a0c36a73f73/volumes" Oct 29 23:45:23.488592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.488577 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d0cbcca-d158-4dbc-88f0-e827b54c630e" path="/var/lib/kubelet/pods/5d0cbcca-d158-4dbc-88f0-e827b54c630e/volumes" Oct 29 23:45:23.488986 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.488972 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca325758-625b-4f60-9b98-9a34cb9c35b7" path="/var/lib/kubelet/pods/ca325758-625b-4f60-9b98-9a34cb9c35b7/volumes" Oct 29 23:45:23.489722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.489696 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cca7fdd3-eac8-49b9-bb6a-a621194a686b" path="/var/lib/kubelet/pods/cca7fdd3-eac8-49b9-bb6a-a621194a686b/volumes" Oct 29 23:45:23.490259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.490241 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2707a1f-9602-475e-83ea-df56bcf01224" path="/var/lib/kubelet/pods/e2707a1f-9602-475e-83ea-df56bcf01224/volumes" Oct 29 23:45:23.490756 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.490739 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142" path="/var/lib/kubelet/pods/f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142/volumes" Oct 29 23:45:23.491058 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.491035 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" Oct 29 23:45:23.492473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.492445 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz" event={"ID":"da38ccbd-b489-4225-a4d1-4502943eb3d4","Type":"ContainerDied","Data":"d957f4b076da0cd4f302cfbeec5db65d09d12242760a0b5d04702124520c7015"} Oct 29 23:45:23.492594 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.492498 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" event={"ID":"4d541e01-8a85-4dd3-801f-9a44798a0ed0","Type":"ContainerDied","Data":"e0f2a532ab1ad303e07631de98c55510aa962e19c6a3d38ba0ab5a2db7f5ca87"} Oct 29 23:45:23.492594 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.492534 5057 scope.go:122] "RemoveContainer" containerID="f360ea40c904cc8df338e27c38f481a55ec35ddfaffd66ce7557560a48650221" Oct 29 23:45:23.492594 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.492586 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b" Oct 29 23:45:23.498856 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.498773 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" event={"ID":"759dfe21-e08a-42ae-a63d-097ada97aa15","Type":"ContainerDied","Data":"cc65a471f994385c4a2484c7c1826fbb65374abe7e5204cf5a7a07268f00bdba"} Oct 29 23:45:23.498985 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.498868 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5" Oct 29 23:45:23.506402 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.506378 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" Oct 29 23:45:23.506518 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.506403 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5" event={"ID":"848ba138-a799-4dab-b704-2ae74e04ac95","Type":"ContainerDied","Data":"7e34d8d69d1287bb4892ee3c4289aa56814e21b1dd64cfd9bd9bc1fcb23baca6"} Oct 29 23:45:23.508308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.508285 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc" event={"ID":"2b0ed631-9976-4f08-8d68-e783428c6a48","Type":"ContainerDied","Data":"8c1218d62e44081e0fa82b7ba30e2cf93b58917925c1a12c4fa7ce48e2c970a7"} Oct 29 23:45:23.508401 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.508356 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc" Oct 29 23:45:23.517663 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.517642 5057 scope.go:122] "RemoveContainer" containerID="664141a556ed144ea4650fd1983c12d08e4732ceb014d63db783b3f64dc759b2" Oct 29 23:45:23.534006 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.533983 5057 scope.go:122] "RemoveContainer" containerID="8bda2e981f761905adeefcc1eca3d785ea88b07eb3804fd604415a071ea0cbfd" Oct 29 23:45:23.552644 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.552619 5057 scope.go:122] "RemoveContainer" containerID="1002b58c271a7d7132fca47eddf28b6aec42181f5f93121cc0c5444274567a1a" Oct 29 23:45:23.565781 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.565752 5057 scope.go:122] "RemoveContainer" containerID="930ec3be00e3078706d3917d8e7778862a2bc5412a4ef8d3c55e72faef56ac34" Oct 29 23:45:23.578273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.578251 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz"] Oct 29 23:45:23.589295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.589266 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e2ac7-29888625-r5mbz"] Oct 29 23:45:23.628147 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.628108 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b"] Oct 29 23:45:23.631925 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.631883 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888625-m4g9b"] Oct 29 23:45:23.658929 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.658891 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc"] Oct 29 23:45:23.668211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.668163 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-37498a-29888625-m55gc"] Oct 29 23:45:23.701266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.701223 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5"] Oct 29 23:45:23.706953 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.706919 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888625-dq7m5"] Oct 29 23:45:23.741336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.741252 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5"] Oct 29 23:45:23.754502 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:23.754466 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2b0108-29888625-glqj5"] Oct 29 23:45:25.502509 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:25.502472 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b0ed631-9976-4f08-8d68-e783428c6a48" path="/var/lib/kubelet/pods/2b0ed631-9976-4f08-8d68-e783428c6a48/volumes" Oct 29 23:45:25.503376 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:25.503351 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d541e01-8a85-4dd3-801f-9a44798a0ed0" path="/var/lib/kubelet/pods/4d541e01-8a85-4dd3-801f-9a44798a0ed0/volumes" Oct 29 23:45:25.504053 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:25.504032 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="759dfe21-e08a-42ae-a63d-097ada97aa15" path="/var/lib/kubelet/pods/759dfe21-e08a-42ae-a63d-097ada97aa15/volumes" Oct 29 23:45:25.505156 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:25.505133 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="848ba138-a799-4dab-b704-2ae74e04ac95" path="/var/lib/kubelet/pods/848ba138-a799-4dab-b704-2ae74e04ac95/volumes" Oct 29 23:45:25.505845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:25.505821 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da38ccbd-b489-4225-a4d1-4502943eb3d4" path="/var/lib/kubelet/pods/da38ccbd-b489-4225-a4d1-4502943eb3d4/volumes" Oct 29 23:45:25.517618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:25.517583 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw"] Oct 29 23:45:25.517869 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:25.517843 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" podUID="b783e0d4-8272-4c3e-b046-0d6a70160fea" containerName="istio-proxy" containerID="cri-o://558dd3cbd60243bb4d7f90e1fd45e441fba0cb26c962d104eaab43a80c6b87be" gracePeriod=30 Oct 29 23:45:28.742112 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.742071 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59"] Oct 29 23:45:28.743006 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.742988 5057 cpu_manager.go:414] "containerMap: removing container" podUID="01fce6be-41f6-4874-b280-6f0422b87a06" containerName="cleanup" Oct 29 23:45:28.743006 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743007 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01fce6be-41f6-4874-b280-6f0422b87a06" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743020 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5d0cbcca-d158-4dbc-88f0-e827b54c630e" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743025 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0cbcca-d158-4dbc-88f0-e827b54c630e" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743039 5057 cpu_manager.go:414] "containerMap: removing container" podUID="2dc6fc01-4438-439e-8106-2a0c36a73f73" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743047 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc6fc01-4438-439e-8106-2a0c36a73f73" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743059 5057 cpu_manager.go:414] "containerMap: removing container" podUID="8496326e-aae6-49fc-be66-4cdeb78ccf74" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743064 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8496326e-aae6-49fc-be66-4cdeb78ccf74" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743072 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e2707a1f-9602-475e-83ea-df56bcf01224" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743077 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2707a1f-9602-475e-83ea-df56bcf01224" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743084 5057 cpu_manager.go:414] "containerMap: removing container" podUID="ca325758-625b-4f60-9b98-9a34cb9c35b7" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743089 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca325758-625b-4f60-9b98-9a34cb9c35b7" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743095 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743100 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743107 5057 cpu_manager.go:414] "containerMap: removing container" podUID="2c26d97b-29bd-4a99-813a-91cd764b3199" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743112 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c26d97b-29bd-4a99-813a-91cd764b3199" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743119 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4d541e01-8a85-4dd3-801f-9a44798a0ed0" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743124 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d541e01-8a85-4dd3-801f-9a44798a0ed0" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743137 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5d0cbcca-d158-4dbc-88f0-e827b54c630e" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743142 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0cbcca-d158-4dbc-88f0-e827b54c630e" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743148 5057 cpu_manager.go:414] "containerMap: removing container" podUID="848ba138-a799-4dab-b704-2ae74e04ac95" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743153 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="848ba138-a799-4dab-b704-2ae74e04ac95" containerName="cleanup" Oct 29 23:45:28.743146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743161 5057 cpu_manager.go:414] "containerMap: removing container" podUID="2b0ed631-9976-4f08-8d68-e783428c6a48" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743166 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b0ed631-9976-4f08-8d68-e783428c6a48" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743172 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1d322efb-9ed7-4868-b1b5-6ac26e0ce34d" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743176 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d322efb-9ed7-4868-b1b5-6ac26e0ce34d" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743182 5057 cpu_manager.go:414] "containerMap: removing container" podUID="da38ccbd-b489-4225-a4d1-4502943eb3d4" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743200 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da38ccbd-b489-4225-a4d1-4502943eb3d4" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743208 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e2707a1f-9602-475e-83ea-df56bcf01224" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743213 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2707a1f-9602-475e-83ea-df56bcf01224" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743224 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1d322efb-9ed7-4868-b1b5-6ac26e0ce34d" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743228 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d322efb-9ed7-4868-b1b5-6ac26e0ce34d" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743236 5057 cpu_manager.go:414] "containerMap: removing container" podUID="ca325758-625b-4f60-9b98-9a34cb9c35b7" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743241 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca325758-625b-4f60-9b98-9a34cb9c35b7" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743247 5057 cpu_manager.go:414] "containerMap: removing container" podUID="da38ccbd-b489-4225-a4d1-4502943eb3d4" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743251 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da38ccbd-b489-4225-a4d1-4502943eb3d4" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743260 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743264 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743271 5057 cpu_manager.go:414] "containerMap: removing container" podUID="759dfe21-e08a-42ae-a63d-097ada97aa15" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743275 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="759dfe21-e08a-42ae-a63d-097ada97aa15" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743283 5057 cpu_manager.go:414] "containerMap: removing container" podUID="2b0ed631-9976-4f08-8d68-e783428c6a48" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743288 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b0ed631-9976-4f08-8d68-e783428c6a48" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743296 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4d541e01-8a85-4dd3-801f-9a44798a0ed0" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743302 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d541e01-8a85-4dd3-801f-9a44798a0ed0" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743309 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5d0cbcca-d158-4dbc-88f0-e827b54c630e" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743313 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0cbcca-d158-4dbc-88f0-e827b54c630e" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743329 5057 cpu_manager.go:414] "containerMap: removing container" podUID="2c26d97b-29bd-4a99-813a-91cd764b3199" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743333 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c26d97b-29bd-4a99-813a-91cd764b3199" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743342 5057 cpu_manager.go:414] "containerMap: removing container" podUID="cca7fdd3-eac8-49b9-bb6a-a621194a686b" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743346 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca7fdd3-eac8-49b9-bb6a-a621194a686b" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743360 5057 cpu_manager.go:414] "containerMap: removing container" podUID="759dfe21-e08a-42ae-a63d-097ada97aa15" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743364 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="759dfe21-e08a-42ae-a63d-097ada97aa15" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743372 5057 cpu_manager.go:414] "containerMap: removing container" podUID="2c26d97b-29bd-4a99-813a-91cd764b3199" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743376 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c26d97b-29bd-4a99-813a-91cd764b3199" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743383 5057 cpu_manager.go:414] "containerMap: removing container" podUID="cca7fdd3-eac8-49b9-bb6a-a621194a686b" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743387 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca7fdd3-eac8-49b9-bb6a-a621194a686b" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743394 5057 cpu_manager.go:414] "containerMap: removing container" podUID="2dc6fc01-4438-439e-8106-2a0c36a73f73" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743398 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc6fc01-4438-439e-8106-2a0c36a73f73" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743405 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4d541e01-8a85-4dd3-801f-9a44798a0ed0" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743409 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d541e01-8a85-4dd3-801f-9a44798a0ed0" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743422 5057 cpu_manager.go:414] "containerMap: removing container" podUID="848ba138-a799-4dab-b704-2ae74e04ac95" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743426 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="848ba138-a799-4dab-b704-2ae74e04ac95" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743433 5057 cpu_manager.go:414] "containerMap: removing container" podUID="01fce6be-41f6-4874-b280-6f0422b87a06" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743438 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01fce6be-41f6-4874-b280-6f0422b87a06" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743445 5057 cpu_manager.go:414] "containerMap: removing container" podUID="759dfe21-e08a-42ae-a63d-097ada97aa15" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743450 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="759dfe21-e08a-42ae-a63d-097ada97aa15" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743580 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="848ba138-a799-4dab-b704-2ae74e04ac95" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743590 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="01fce6be-41f6-4874-b280-6f0422b87a06" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743599 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c26d97b-29bd-4a99-813a-91cd764b3199" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743604 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca7fdd3-eac8-49b9-bb6a-a621194a686b" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743611 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="da38ccbd-b489-4225-a4d1-4502943eb3d4" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743616 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d322efb-9ed7-4868-b1b5-6ac26e0ce34d" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743621 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="759dfe21-e08a-42ae-a63d-097ada97aa15" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743626 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c26d97b-29bd-4a99-813a-91cd764b3199" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743634 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca7fdd3-eac8-49b9-bb6a-a621194a686b" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743641 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc6fc01-4438-439e-8106-2a0c36a73f73" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743646 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc6fc01-4438-439e-8106-2a0c36a73f73" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743653 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d541e01-8a85-4dd3-801f-9a44798a0ed0" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743659 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="01fce6be-41f6-4874-b280-6f0422b87a06" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743664 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="759dfe21-e08a-42ae-a63d-097ada97aa15" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743669 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b0ed631-9976-4f08-8d68-e783428c6a48" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743673 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="8496326e-aae6-49fc-be66-4cdeb78ccf74" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743678 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca7fdd3-eac8-49b9-bb6a-a621194a686b" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743686 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="01fce6be-41f6-4874-b280-6f0422b87a06" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743691 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0cbcca-d158-4dbc-88f0-e827b54c630e" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743696 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc6fc01-4438-439e-8106-2a0c36a73f73" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743702 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="8496326e-aae6-49fc-be66-4cdeb78ccf74" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743708 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2707a1f-9602-475e-83ea-df56bcf01224" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743714 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca325758-625b-4f60-9b98-9a34cb9c35b7" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743718 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743724 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d541e01-8a85-4dd3-801f-9a44798a0ed0" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743731 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0cbcca-d158-4dbc-88f0-e827b54c630e" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743736 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="848ba138-a799-4dab-b704-2ae74e04ac95" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743740 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b0ed631-9976-4f08-8d68-e783428c6a48" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743745 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d322efb-9ed7-4868-b1b5-6ac26e0ce34d" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743750 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2707a1f-9602-475e-83ea-df56bcf01224" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743756 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743762 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2707a1f-9602-475e-83ea-df56bcf01224" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743769 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca325758-625b-4f60-9b98-9a34cb9c35b7" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743775 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="da38ccbd-b489-4225-a4d1-4502943eb3d4" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743782 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="759dfe21-e08a-42ae-a63d-097ada97aa15" containerName="cleanup" Oct 29 23:45:28.744138 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.743788 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca325758-625b-4f60-9b98-9a34cb9c35b7" containerName="cleanup" Oct 29 23:45:28.750357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.750334 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.765906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.765839 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf"] Oct 29 23:45:28.767103 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767085 5057 cpu_manager.go:414] "containerMap: removing container" podUID="01fce6be-41f6-4874-b280-6f0422b87a06" containerName="cleanup" Oct 29 23:45:28.767181 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767105 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01fce6be-41f6-4874-b280-6f0422b87a06" containerName="cleanup" Oct 29 23:45:28.767181 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767127 5057 cpu_manager.go:414] "containerMap: removing container" podUID="2dc6fc01-4438-439e-8106-2a0c36a73f73" containerName="cleanup" Oct 29 23:45:28.767181 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767135 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc6fc01-4438-439e-8106-2a0c36a73f73" containerName="cleanup" Oct 29 23:45:28.767181 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767149 5057 cpu_manager.go:414] "containerMap: removing container" podUID="8496326e-aae6-49fc-be66-4cdeb78ccf74" containerName="cleanup" Oct 29 23:45:28.767181 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767157 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8496326e-aae6-49fc-be66-4cdeb78ccf74" containerName="cleanup" Oct 29 23:45:28.767181 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767170 5057 cpu_manager.go:414] "containerMap: removing container" podUID="ca325758-625b-4f60-9b98-9a34cb9c35b7" containerName="cleanup" Oct 29 23:45:28.767181 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767177 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca325758-625b-4f60-9b98-9a34cb9c35b7" containerName="cleanup" Oct 29 23:45:28.767442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767202 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142" containerName="cleanup" Oct 29 23:45:28.767442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767210 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142" containerName="cleanup" Oct 29 23:45:28.767442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767241 5057 cpu_manager.go:414] "containerMap: removing container" podUID="848ba138-a799-4dab-b704-2ae74e04ac95" containerName="cleanup" Oct 29 23:45:28.767442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767247 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="848ba138-a799-4dab-b704-2ae74e04ac95" containerName="cleanup" Oct 29 23:45:28.767442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767261 5057 cpu_manager.go:414] "containerMap: removing container" podUID="da38ccbd-b489-4225-a4d1-4502943eb3d4" containerName="cleanup" Oct 29 23:45:28.767442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767268 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da38ccbd-b489-4225-a4d1-4502943eb3d4" containerName="cleanup" Oct 29 23:45:28.767442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767282 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e2707a1f-9602-475e-83ea-df56bcf01224" containerName="cleanup" Oct 29 23:45:28.767442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767288 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2707a1f-9602-475e-83ea-df56bcf01224" containerName="cleanup" Oct 29 23:45:28.767442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767310 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1d322efb-9ed7-4868-b1b5-6ac26e0ce34d" containerName="cleanup" Oct 29 23:45:28.767442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767316 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d322efb-9ed7-4868-b1b5-6ac26e0ce34d" containerName="cleanup" Oct 29 23:45:28.767442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767341 5057 cpu_manager.go:414] "containerMap: removing container" podUID="2b0ed631-9976-4f08-8d68-e783428c6a48" containerName="cleanup" Oct 29 23:45:28.767442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767348 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b0ed631-9976-4f08-8d68-e783428c6a48" containerName="cleanup" Oct 29 23:45:28.767442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767392 5057 cpu_manager.go:414] "containerMap: removing container" podUID="cca7fdd3-eac8-49b9-bb6a-a621194a686b" containerName="cleanup" Oct 29 23:45:28.767442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767398 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca7fdd3-eac8-49b9-bb6a-a621194a686b" containerName="cleanup" Oct 29 23:45:28.767862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767600 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0cbcca-d158-4dbc-88f0-e827b54c630e" containerName="cleanup" Oct 29 23:45:28.767862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767615 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d541e01-8a85-4dd3-801f-9a44798a0ed0" containerName="cleanup" Oct 29 23:45:28.767862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767628 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="848ba138-a799-4dab-b704-2ae74e04ac95" containerName="cleanup" Oct 29 23:45:28.767862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767641 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b0ed631-9976-4f08-8d68-e783428c6a48" containerName="cleanup" Oct 29 23:45:28.767862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767650 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d322efb-9ed7-4868-b1b5-6ac26e0ce34d" containerName="cleanup" Oct 29 23:45:28.767862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767662 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b4ab25-72c1-4da3-bbdf-fb1e3ea7f142" containerName="cleanup" Oct 29 23:45:28.767862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767673 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="da38ccbd-b489-4225-a4d1-4502943eb3d4" containerName="cleanup" Oct 29 23:45:28.767862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.767690 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c26d97b-29bd-4a99-813a-91cd764b3199" containerName="cleanup" Oct 29 23:45:28.770230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.770176 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-825472-openshift-default-dockercfg-vh6rb\"" Oct 29 23:45:28.772333 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.772309 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.777036 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.777007 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgltd\" (UniqueName: \"kubernetes.io/projected/87c126a1-a9e4-4aa9-afe9-78727290d709-kube-api-access-mgltd\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.777141 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.777092 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.777141 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.777137 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.777285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.777160 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.777285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.777204 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c126a1-a9e4-4aa9-afe9-78727290d709-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.777362 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.777302 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.777362 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.777330 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.777436 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.777360 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.777436 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.777377 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.782326 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.782300 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59"] Oct 29 23:45:28.818325 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.800777 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf"] Oct 29 23:45:28.879532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.879483 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.879698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.879559 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.879698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.879599 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.879698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.879634 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c126a1-a9e4-4aa9-afe9-78727290d709-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.879698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.879672 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.879961 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.879734 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.879961 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.879870 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.879961 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.879937 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.880142 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.879968 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.880142 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.879993 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.880142 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.880004 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.880142 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.880022 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.880142 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.880098 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfee661-d5d7-4b6f-8691-aec67412f360-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.880454 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.880160 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.880454 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.880168 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.880454 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.880266 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kss52\" (UniqueName: \"kubernetes.io/projected/abfee661-d5d7-4b6f-8691-aec67412f360-kube-api-access-kss52\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.880454 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.880405 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.880713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.880506 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.880713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.880530 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfee661-d5d7-4b6f-8691-aec67412f360-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.880713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.880563 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.880713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.880587 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c126a1-a9e4-4aa9-afe9-78727290d709-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.880713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.880644 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfee661-d5d7-4b6f-8691-aec67412f360-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.880884 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.880726 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgltd\" (UniqueName: \"kubernetes.io/projected/87c126a1-a9e4-4aa9-afe9-78727290d709-kube-api-access-mgltd\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.882256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.882233 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.882941 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.882923 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.904356 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.904330 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.905316 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.905286 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgltd\" (UniqueName: \"kubernetes.io/projected/87c126a1-a9e4-4aa9-afe9-78727290d709-kube-api-access-mgltd\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:28.982533 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.982492 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.982746 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.982652 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfee661-d5d7-4b6f-8691-aec67412f360-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.982746 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.982712 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.982873 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.982742 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kss52\" (UniqueName: \"kubernetes.io/projected/abfee661-d5d7-4b6f-8691-aec67412f360-kube-api-access-kss52\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.982873 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.982817 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.982979 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.982873 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.982979 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.982890 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfee661-d5d7-4b6f-8691-aec67412f360-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.982979 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.982940 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.983273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.982986 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfee661-d5d7-4b6f-8691-aec67412f360-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.983273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.983077 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.983273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.983158 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.983273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.983231 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.983579 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.983552 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.983757 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.983734 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfee661-d5d7-4b6f-8691-aec67412f360-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.985168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.985146 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:28.985382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:28.985366 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfee661-d5d7-4b6f-8691-aec67412f360-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:29.008337 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:29.008267 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfee661-d5d7-4b6f-8691-aec67412f360-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:29.008453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:29.008420 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kss52\" (UniqueName: \"kubernetes.io/projected/abfee661-d5d7-4b6f-8691-aec67412f360-kube-api-access-kss52\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:29.068849 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:29.068809 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:29.099703 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:29.099676 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:29.412929 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:29.412855 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59"] Oct 29 23:45:29.426099 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:45:29.426066 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c126a1_a9e4_4aa9_afe9_78727290d709.slice/crio-61fe47447b0a58d7a7c609e11c4ce843fa164bbf5af4936accda06b230eb9747 WatchSource:0}: Error finding container 61fe47447b0a58d7a7c609e11c4ce843fa164bbf5af4936accda06b230eb9747: Status 404 returned error can't find the container with id 61fe47447b0a58d7a7c609e11c4ce843fa164bbf5af4936accda06b230eb9747 Oct 29 23:45:29.429915 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:29.429875 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:45:29.430035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:29.429967 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:45:29.430035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:29.429997 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:45:29.618058 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:29.618008 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:45:29.651221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:29.641365 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:45:29.651221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:29.641625 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:45:30.116482 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:30.116434 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf"] Oct 29 23:45:30.257577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:30.257534 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" event={"ID":"abfee661-d5d7-4b6f-8691-aec67412f360","Type":"ContainerStarted","Data":"09145a9b9f9fc33ef4bcaf6e27d17f3fd30e7bafaa7a539c8822bd3abe29c2b7"} Oct 29 23:45:30.266668 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:30.266582 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" event={"ID":"87c126a1-a9e4-4aa9-afe9-78727290d709","Type":"ContainerStarted","Data":"61fe47447b0a58d7a7c609e11c4ce843fa164bbf5af4936accda06b230eb9747"} Oct 29 23:45:31.126600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.126570 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:45:31.175817 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.175777 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-data\") pod \"b783e0d4-8272-4c3e-b046-0d6a70160fea\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " Oct 29 23:45:31.175994 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.175837 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-podinfo\") pod \"b783e0d4-8272-4c3e-b046-0d6a70160fea\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " Oct 29 23:45:31.175994 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.175879 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-envoy\") pod \"b783e0d4-8272-4c3e-b046-0d6a70160fea\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " Oct 29 23:45:31.175994 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.175971 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b783e0d4-8272-4c3e-b046-0d6a70160fea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b783e0d4-8272-4c3e-b046-0d6a70160fea-istiod-ca-cert\") pod \"b783e0d4-8272-4c3e-b046-0d6a70160fea\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " Oct 29 23:45:31.176205 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.176018 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-credential-socket\") pod \"b783e0d4-8272-4c3e-b046-0d6a70160fea\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " Oct 29 23:45:31.176205 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.176082 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-workload-certs\") pod \"b783e0d4-8272-4c3e-b046-0d6a70160fea\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " Oct 29 23:45:31.176205 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.176120 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-token\" (UniqueName: \"kubernetes.io/projected/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-token\") pod \"b783e0d4-8272-4c3e-b046-0d6a70160fea\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " Oct 29 23:45:31.176376 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.176313 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-workload-socket\") pod \"b783e0d4-8272-4c3e-b046-0d6a70160fea\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " Oct 29 23:45:31.176485 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.176464 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b783e0d4-8272-4c3e-b046-0d6a70160fea-kube-api-access-h7tjn\" (UniqueName: \"kubernetes.io/projected/b783e0d4-8272-4c3e-b046-0d6a70160fea-kube-api-access-h7tjn\") pod \"b783e0d4-8272-4c3e-b046-0d6a70160fea\" (UID: \"b783e0d4-8272-4c3e-b046-0d6a70160fea\") " Oct 29 23:45:31.178595 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.178554 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-envoy" pod "b783e0d4-8272-4c3e-b046-0d6a70160fea" (UID: "b783e0d4-8272-4c3e-b046-0d6a70160fea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:31.178797 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.178777 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-data" pod "b783e0d4-8272-4c3e-b046-0d6a70160fea" (UID: "b783e0d4-8272-4c3e-b046-0d6a70160fea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:31.179796 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.179766 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-workload-certs" pod "b783e0d4-8272-4c3e-b046-0d6a70160fea" (UID: "b783e0d4-8272-4c3e-b046-0d6a70160fea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:31.180113 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.180088 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b783e0d4-8272-4c3e-b046-0d6a70160fea-istiod-ca-cert" pod "b783e0d4-8272-4c3e-b046-0d6a70160fea" (UID: "b783e0d4-8272-4c3e-b046-0d6a70160fea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:31.180418 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.180393 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-credential-socket" pod "b783e0d4-8272-4c3e-b046-0d6a70160fea" (UID: "b783e0d4-8272-4c3e-b046-0d6a70160fea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:31.181081 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.181054 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-podinfo" pod "b783e0d4-8272-4c3e-b046-0d6a70160fea" (UID: "b783e0d4-8272-4c3e-b046-0d6a70160fea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:45:31.181313 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.181291 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-workload-socket" pod "b783e0d4-8272-4c3e-b046-0d6a70160fea" (UID: "b783e0d4-8272-4c3e-b046-0d6a70160fea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:31.181442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.181415 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b783e0d4-8272-4c3e-b046-0d6a70160fea-kube-api-access-h7tjn" pod "b783e0d4-8272-4c3e-b046-0d6a70160fea" (UID: "b783e0d4-8272-4c3e-b046-0d6a70160fea"). InnerVolumeSpecName "kube-api-access-h7tjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:31.182506 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.182482 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-token" pod "b783e0d4-8272-4c3e-b046-0d6a70160fea" (UID: "b783e0d4-8272-4c3e-b046-0d6a70160fea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:31.279038 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.279002 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:31.279038 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.279039 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:31.279323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.279053 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:31.279323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.279066 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b783e0d4-8272-4c3e-b046-0d6a70160fea-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:31.279323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.279078 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:31.279323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.279092 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:31.279323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.279107 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b783e0d4-8272-4c3e-b046-0d6a70160fea-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:31.279323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.279120 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b783e0d4-8272-4c3e-b046-0d6a70160fea-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:31.279323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.279132 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7tjn\" (UniqueName: \"kubernetes.io/projected/b783e0d4-8272-4c3e-b046-0d6a70160fea-kube-api-access-h7tjn\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:31.280466 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.280428 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" event={"ID":"abfee661-d5d7-4b6f-8691-aec67412f360","Type":"ContainerStarted","Data":"fc8617404d84937ea24f36c0f5149954dddcf3ea6f8341b050d1593645a3130f"} Oct 29 23:45:31.282281 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.282253 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" event={"ID":"87c126a1-a9e4-4aa9-afe9-78727290d709","Type":"ContainerStarted","Data":"c535f2396cc502d5ed11aa1ede4ac0a5350729c5e4228b35af44be4f1257e76b"} Oct 29 23:45:31.284339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.284314 5057 generic.go:356] "Generic (PLEG): container finished" podID="b783e0d4-8272-4c3e-b046-0d6a70160fea" containerID="558dd3cbd60243bb4d7f90e1fd45e441fba0cb26c962d104eaab43a80c6b87be" exitCode=0 Oct 29 23:45:31.284509 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.284366 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" Oct 29 23:45:31.284612 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.284396 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" event={"ID":"b783e0d4-8272-4c3e-b046-0d6a70160fea","Type":"ContainerDied","Data":"558dd3cbd60243bb4d7f90e1fd45e441fba0cb26c962d104eaab43a80c6b87be"} Oct 29 23:45:31.284685 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.284649 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw" event={"ID":"b783e0d4-8272-4c3e-b046-0d6a70160fea","Type":"ContainerDied","Data":"b64ddaf5898463fb835a295d3914a0b0d58029a059c60f5109d4abf91df3346f"} Oct 29 23:45:31.284685 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.284679 5057 scope.go:122] "RemoveContainer" containerID="558dd3cbd60243bb4d7f90e1fd45e441fba0cb26c962d104eaab43a80c6b87be" Oct 29 23:45:31.300101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.300078 5057 scope.go:122] "RemoveContainer" containerID="a9df19ba5de2e880e94d51add5dbaffa926cb5932164ed84a8b74f4607ad03d0" Oct 29 23:45:31.311105 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.311087 5057 scope.go:122] "RemoveContainer" containerID="558dd3cbd60243bb4d7f90e1fd45e441fba0cb26c962d104eaab43a80c6b87be" Oct 29 23:45:31.311384 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:31.311362 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"558dd3cbd60243bb4d7f90e1fd45e441fba0cb26c962d104eaab43a80c6b87be\": container with ID starting with 558dd3cbd60243bb4d7f90e1fd45e441fba0cb26c962d104eaab43a80c6b87be not found: ID does not exist" containerID="558dd3cbd60243bb4d7f90e1fd45e441fba0cb26c962d104eaab43a80c6b87be" Oct 29 23:45:31.311457 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.311390 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"558dd3cbd60243bb4d7f90e1fd45e441fba0cb26c962d104eaab43a80c6b87be"} err="failed to get container status \"558dd3cbd60243bb4d7f90e1fd45e441fba0cb26c962d104eaab43a80c6b87be\": rpc error: code = NotFound desc = could not find container \"558dd3cbd60243bb4d7f90e1fd45e441fba0cb26c962d104eaab43a80c6b87be\": container with ID starting with 558dd3cbd60243bb4d7f90e1fd45e441fba0cb26c962d104eaab43a80c6b87be not found: ID does not exist" Oct 29 23:45:31.311457 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.311407 5057 scope.go:122] "RemoveContainer" containerID="a9df19ba5de2e880e94d51add5dbaffa926cb5932164ed84a8b74f4607ad03d0" Oct 29 23:45:31.311655 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:31.311639 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9df19ba5de2e880e94d51add5dbaffa926cb5932164ed84a8b74f4607ad03d0\": container with ID starting with a9df19ba5de2e880e94d51add5dbaffa926cb5932164ed84a8b74f4607ad03d0 not found: ID does not exist" containerID="a9df19ba5de2e880e94d51add5dbaffa926cb5932164ed84a8b74f4607ad03d0" Oct 29 23:45:31.311694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.311660 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9df19ba5de2e880e94d51add5dbaffa926cb5932164ed84a8b74f4607ad03d0"} err="failed to get container status \"a9df19ba5de2e880e94d51add5dbaffa926cb5932164ed84a8b74f4607ad03d0\": rpc error: code = NotFound desc = could not find container \"a9df19ba5de2e880e94d51add5dbaffa926cb5932164ed84a8b74f4607ad03d0\": container with ID starting with a9df19ba5de2e880e94d51add5dbaffa926cb5932164ed84a8b74f4607ad03d0 not found: ID does not exist" Oct 29 23:45:31.326252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.326163 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" podStartSLOduration=3.326148736 podStartE2EDuration="3.326148736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:31.324140414 +0000 UTC m=+3777.975638944" watchObservedRunningTime="2026-10-29 23:45:31.326148736 +0000 UTC m=+3777.977647266" Oct 29 23:45:31.375775 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.375679 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" podStartSLOduration=3.375665139 podStartE2EDuration="3.375665139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:31.37047099 +0000 UTC m=+3778.021969534" watchObservedRunningTime="2026-10-29 23:45:31.375665139 +0000 UTC m=+3778.027163669" Oct 29 23:45:31.418137 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.418106 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw"] Oct 29 23:45:31.422919 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.422889 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-n64dw"] Oct 29 23:45:31.490197 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:31.485152 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b783e0d4-8272-4c3e-b046-0d6a70160fea" path="/var/lib/kubelet/pods/b783e0d4-8272-4c3e-b046-0d6a70160fea/volumes" Oct 29 23:45:32.088802 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:32.086077 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:32.101213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:32.096328 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body= Oct 29 23:45:32.101213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:32.096401 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" podUID="87c126a1-a9e4-4aa9-afe9-78727290d709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" Oct 29 23:45:32.120214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:32.111361 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:32.120214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:32.111964 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body= Oct 29 23:45:32.120214 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:32.112020 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" podUID="abfee661-d5d7-4b6f-8691-aec67412f360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" Oct 29 23:45:33.072210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:33.069635 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body= Oct 29 23:45:33.072210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:33.069698 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" podUID="87c126a1-a9e4-4aa9-afe9-78727290d709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" Oct 29 23:45:33.108212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:33.100545 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body= Oct 29 23:45:33.108212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:33.100621 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" podUID="abfee661-d5d7-4b6f-8691-aec67412f360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" Oct 29 23:45:33.990210 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:33.986412 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:45:34.079211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:34.078727 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body= Oct 29 23:45:34.079211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:34.078791 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" podUID="87c126a1-a9e4-4aa9-afe9-78727290d709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" Oct 29 23:45:34.100703 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:34.100667 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body= Oct 29 23:45:34.100878 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:34.100718 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" podUID="abfee661-d5d7-4b6f-8691-aec67412f360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" Oct 29 23:45:35.076210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:35.075624 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body= Oct 29 23:45:35.076210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:35.075690 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" podUID="87c126a1-a9e4-4aa9-afe9-78727290d709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" Oct 29 23:45:35.100393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:35.100360 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body= Oct 29 23:45:35.100753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:35.100418 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" podUID="abfee661-d5d7-4b6f-8691-aec67412f360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" Oct 29 23:45:36.073211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:36.070284 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body= Oct 29 23:45:36.073211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:36.070341 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" podUID="87c126a1-a9e4-4aa9-afe9-78727290d709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" Oct 29 23:45:36.101207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:36.100840 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body= Oct 29 23:45:36.101207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:36.100900 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" podUID="abfee661-d5d7-4b6f-8691-aec67412f360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" Oct 29 23:45:37.107223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:37.107009 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body= Oct 29 23:45:37.107223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:37.107067 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" podUID="abfee661-d5d7-4b6f-8691-aec67412f360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" Oct 29 23:45:37.108271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:37.108241 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body= Oct 29 23:45:37.108431 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:37.108411 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" podUID="87c126a1-a9e4-4aa9-afe9-78727290d709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" Oct 29 23:45:38.104168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:38.104130 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body= Oct 29 23:45:38.104440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:38.104411 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" podUID="abfee661-d5d7-4b6f-8691-aec67412f360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" Oct 29 23:45:38.298398 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:38.297737 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:38.312021 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:38.311990 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:38.313912 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:38.313881 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7"] Oct 29 23:45:38.314405 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:38.314356 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" podUID="b7893cca-8dc6-4395-bc25-452bcd512dcd" containerName="istio-proxy" containerID="cri-o://be55e009b4bffd638e8c709354e3ba8b21debdc2996d44c63e9e6a76d00920b6" gracePeriod=30 Oct 29 23:45:38.372043 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:38.361464 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:45:39.106056 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:39.106026 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:39.106416 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:39.106396 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:39.107053 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:39.107034 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:45:43.929895 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:43.929863 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:45:44.019246 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.019209 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-workload-socket\") pod \"b7893cca-8dc6-4395-bc25-452bcd512dcd\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " Oct 29 23:45:44.019415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.019289 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-podinfo\") pod \"b7893cca-8dc6-4395-bc25-452bcd512dcd\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " Oct 29 23:45:44.019512 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.019414 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-data\") pod \"b7893cca-8dc6-4395-bc25-452bcd512dcd\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " Oct 29 23:45:44.019512 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.019451 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-token\" (UniqueName: \"kubernetes.io/projected/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-token\") pod \"b7893cca-8dc6-4395-bc25-452bcd512dcd\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " Oct 29 23:45:44.019512 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.019503 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-envoy\") pod \"b7893cca-8dc6-4395-bc25-452bcd512dcd\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " Oct 29 23:45:44.019691 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.019614 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-workload-socket" pod "b7893cca-8dc6-4395-bc25-452bcd512dcd" (UID: "b7893cca-8dc6-4395-bc25-452bcd512dcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:44.019691 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.019632 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7893cca-8dc6-4395-bc25-452bcd512dcd-kube-api-access-jppz5\" (UniqueName: \"kubernetes.io/projected/b7893cca-8dc6-4395-bc25-452bcd512dcd-kube-api-access-jppz5\") pod \"b7893cca-8dc6-4395-bc25-452bcd512dcd\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " Oct 29 23:45:44.019691 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.019656 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-workload-certs\") pod \"b7893cca-8dc6-4395-bc25-452bcd512dcd\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " Oct 29 23:45:44.019691 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.019687 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-credential-socket\") pod \"b7893cca-8dc6-4395-bc25-452bcd512dcd\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " Oct 29 23:45:44.019905 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.019722 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7893cca-8dc6-4395-bc25-452bcd512dcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7893cca-8dc6-4395-bc25-452bcd512dcd-istiod-ca-cert\") pod \"b7893cca-8dc6-4395-bc25-452bcd512dcd\" (UID: \"b7893cca-8dc6-4395-bc25-452bcd512dcd\") " Oct 29 23:45:44.020404 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.020380 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:44.021028 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.021003 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7893cca-8dc6-4395-bc25-452bcd512dcd-istiod-ca-cert" pod "b7893cca-8dc6-4395-bc25-452bcd512dcd" (UID: "b7893cca-8dc6-4395-bc25-452bcd512dcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:44.021402 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.021377 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-workload-certs" pod "b7893cca-8dc6-4395-bc25-452bcd512dcd" (UID: "b7893cca-8dc6-4395-bc25-452bcd512dcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:44.021926 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.021901 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-data" pod "b7893cca-8dc6-4395-bc25-452bcd512dcd" (UID: "b7893cca-8dc6-4395-bc25-452bcd512dcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:44.022165 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.022145 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-credential-socket" pod "b7893cca-8dc6-4395-bc25-452bcd512dcd" (UID: "b7893cca-8dc6-4395-bc25-452bcd512dcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:44.022326 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.022288 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-token" pod "b7893cca-8dc6-4395-bc25-452bcd512dcd" (UID: "b7893cca-8dc6-4395-bc25-452bcd512dcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:44.024651 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.024625 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-podinfo" pod "b7893cca-8dc6-4395-bc25-452bcd512dcd" (UID: "b7893cca-8dc6-4395-bc25-452bcd512dcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:45:44.024740 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.024649 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-envoy" pod "b7893cca-8dc6-4395-bc25-452bcd512dcd" (UID: "b7893cca-8dc6-4395-bc25-452bcd512dcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:44.024740 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.024686 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7893cca-8dc6-4395-bc25-452bcd512dcd-kube-api-access-jppz5" pod "b7893cca-8dc6-4395-bc25-452bcd512dcd" (UID: "b7893cca-8dc6-4395-bc25-452bcd512dcd"). InnerVolumeSpecName "kube-api-access-jppz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:44.122819 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.122733 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:44.122819 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.122764 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:44.122819 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.122775 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:44.122819 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.122784 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jppz5\" (UniqueName: \"kubernetes.io/projected/b7893cca-8dc6-4395-bc25-452bcd512dcd-kube-api-access-jppz5\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:44.122819 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.122794 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:44.122819 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.122803 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7893cca-8dc6-4395-bc25-452bcd512dcd-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:44.122819 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.122811 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7893cca-8dc6-4395-bc25-452bcd512dcd-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:44.122819 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.122821 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7893cca-8dc6-4395-bc25-452bcd512dcd-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:44.576211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.569634 5057 generic.go:356] "Generic (PLEG): container finished" podID="b7893cca-8dc6-4395-bc25-452bcd512dcd" containerID="be55e009b4bffd638e8c709354e3ba8b21debdc2996d44c63e9e6a76d00920b6" exitCode=0 Oct 29 23:45:44.576211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.569856 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" event={"ID":"b7893cca-8dc6-4395-bc25-452bcd512dcd","Type":"ContainerDied","Data":"be55e009b4bffd638e8c709354e3ba8b21debdc2996d44c63e9e6a76d00920b6"} Oct 29 23:45:44.576211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.569910 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" event={"ID":"b7893cca-8dc6-4395-bc25-452bcd512dcd","Type":"ContainerDied","Data":"7bdffc0fcd5bf3e94ba171a26ad0431f47cf7ab7e2f5a2f6c60f9ca58ba3e569"} Oct 29 23:45:44.576211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.569936 5057 scope.go:122] "RemoveContainer" containerID="be55e009b4bffd638e8c709354e3ba8b21debdc2996d44c63e9e6a76d00920b6" Oct 29 23:45:44.576211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.570139 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7" Oct 29 23:45:44.871308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.871244 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7"] Oct 29 23:45:44.896230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.894394 5057 scope.go:122] "RemoveContainer" containerID="98c68d250dbc7aaad45f78afa67d857bb15b249353ccca9a8b33c33ac1f9aa66" Oct 29 23:45:44.912235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.905134 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e10975-openshift-default-74f4c85b85-frqh7"] Oct 29 23:45:44.941768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.941741 5057 scope.go:122] "RemoveContainer" containerID="be55e009b4bffd638e8c709354e3ba8b21debdc2996d44c63e9e6a76d00920b6" Oct 29 23:45:44.942259 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:44.942231 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be55e009b4bffd638e8c709354e3ba8b21debdc2996d44c63e9e6a76d00920b6\": container with ID starting with be55e009b4bffd638e8c709354e3ba8b21debdc2996d44c63e9e6a76d00920b6 not found: ID does not exist" containerID="be55e009b4bffd638e8c709354e3ba8b21debdc2996d44c63e9e6a76d00920b6" Oct 29 23:45:44.942347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.942283 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be55e009b4bffd638e8c709354e3ba8b21debdc2996d44c63e9e6a76d00920b6"} err="failed to get container status \"be55e009b4bffd638e8c709354e3ba8b21debdc2996d44c63e9e6a76d00920b6\": rpc error: code = NotFound desc = could not find container \"be55e009b4bffd638e8c709354e3ba8b21debdc2996d44c63e9e6a76d00920b6\": container with ID starting with be55e009b4bffd638e8c709354e3ba8b21debdc2996d44c63e9e6a76d00920b6 not found: ID does not exist" Oct 29 23:45:44.942347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.942310 5057 scope.go:122] "RemoveContainer" containerID="98c68d250dbc7aaad45f78afa67d857bb15b249353ccca9a8b33c33ac1f9aa66" Oct 29 23:45:44.942707 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:44.942682 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98c68d250dbc7aaad45f78afa67d857bb15b249353ccca9a8b33c33ac1f9aa66\": container with ID starting with 98c68d250dbc7aaad45f78afa67d857bb15b249353ccca9a8b33c33ac1f9aa66 not found: ID does not exist" containerID="98c68d250dbc7aaad45f78afa67d857bb15b249353ccca9a8b33c33ac1f9aa66" Oct 29 23:45:44.942832 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:44.942812 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c68d250dbc7aaad45f78afa67d857bb15b249353ccca9a8b33c33ac1f9aa66"} err="failed to get container status \"98c68d250dbc7aaad45f78afa67d857bb15b249353ccca9a8b33c33ac1f9aa66\": rpc error: code = NotFound desc = could not find container \"98c68d250dbc7aaad45f78afa67d857bb15b249353ccca9a8b33c33ac1f9aa66\": container with ID starting with 98c68d250dbc7aaad45f78afa67d857bb15b249353ccca9a8b33c33ac1f9aa66 not found: ID does not exist" Oct 29 23:45:45.516493 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:45.516454 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7893cca-8dc6-4395-bc25-452bcd512dcd" path="/var/lib/kubelet/pods/b7893cca-8dc6-4395-bc25-452bcd512dcd/volumes" Oct 29 23:45:52.416481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:52.410351 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg"] Oct 29 23:45:52.416481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:52.410623 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" podUID="7dfdb32d-7346-42ac-a5e6-3081f0895a8e" containerName="istio-proxy" containerID="cri-o://5753df527098f71141bbba9f5aa3786eb655ddf1672fc0c868586eea8679fc75" gracePeriod=30 Oct 29 23:45:52.577211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:52.573371 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:45:52.577211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:52.573453 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" podUID="7dfdb32d-7346-42ac-a5e6-3081f0895a8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:45:57.723978 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.723944 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:45:57.762496 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.762463 5057 generic.go:356] "Generic (PLEG): container finished" podID="7dfdb32d-7346-42ac-a5e6-3081f0895a8e" containerID="5753df527098f71141bbba9f5aa3786eb655ddf1672fc0c868586eea8679fc75" exitCode=0 Oct 29 23:45:57.762682 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.762509 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" Oct 29 23:45:57.762682 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.762547 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" event={"ID":"7dfdb32d-7346-42ac-a5e6-3081f0895a8e","Type":"ContainerDied","Data":"5753df527098f71141bbba9f5aa3786eb655ddf1672fc0c868586eea8679fc75"} Oct 29 23:45:57.762682 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.762605 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg" event={"ID":"7dfdb32d-7346-42ac-a5e6-3081f0895a8e","Type":"ContainerDied","Data":"6bb1b6c5233c10388ddaba2fc2018981a8ce02c727bcbaa2ebc66d33bf03e543"} Oct 29 23:45:57.762682 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.762636 5057 scope.go:122] "RemoveContainer" containerID="5753df527098f71141bbba9f5aa3786eb655ddf1672fc0c868586eea8679fc75" Oct 29 23:45:57.776737 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.776718 5057 scope.go:122] "RemoveContainer" containerID="d2768bcf76f84f06ec88d3a3680e42b3d04598024460cfffcfc7be28f3f8a63b" Oct 29 23:45:57.792298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.791732 5057 scope.go:122] "RemoveContainer" containerID="5753df527098f71141bbba9f5aa3786eb655ddf1672fc0c868586eea8679fc75" Oct 29 23:45:57.792298 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:57.792155 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5753df527098f71141bbba9f5aa3786eb655ddf1672fc0c868586eea8679fc75\": container with ID starting with 5753df527098f71141bbba9f5aa3786eb655ddf1672fc0c868586eea8679fc75 not found: ID does not exist" containerID="5753df527098f71141bbba9f5aa3786eb655ddf1672fc0c868586eea8679fc75" Oct 29 23:45:57.792298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.792202 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5753df527098f71141bbba9f5aa3786eb655ddf1672fc0c868586eea8679fc75"} err="failed to get container status \"5753df527098f71141bbba9f5aa3786eb655ddf1672fc0c868586eea8679fc75\": rpc error: code = NotFound desc = could not find container \"5753df527098f71141bbba9f5aa3786eb655ddf1672fc0c868586eea8679fc75\": container with ID starting with 5753df527098f71141bbba9f5aa3786eb655ddf1672fc0c868586eea8679fc75 not found: ID does not exist" Oct 29 23:45:57.792298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.792227 5057 scope.go:122] "RemoveContainer" containerID="d2768bcf76f84f06ec88d3a3680e42b3d04598024460cfffcfc7be28f3f8a63b" Oct 29 23:45:57.792758 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:45:57.792579 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2768bcf76f84f06ec88d3a3680e42b3d04598024460cfffcfc7be28f3f8a63b\": container with ID starting with d2768bcf76f84f06ec88d3a3680e42b3d04598024460cfffcfc7be28f3f8a63b not found: ID does not exist" containerID="d2768bcf76f84f06ec88d3a3680e42b3d04598024460cfffcfc7be28f3f8a63b" Oct 29 23:45:57.792758 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.792612 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2768bcf76f84f06ec88d3a3680e42b3d04598024460cfffcfc7be28f3f8a63b"} err="failed to get container status \"d2768bcf76f84f06ec88d3a3680e42b3d04598024460cfffcfc7be28f3f8a63b\": rpc error: code = NotFound desc = could not find container \"d2768bcf76f84f06ec88d3a3680e42b3d04598024460cfffcfc7be28f3f8a63b\": container with ID starting with d2768bcf76f84f06ec88d3a3680e42b3d04598024460cfffcfc7be28f3f8a63b not found: ID does not exist" Oct 29 23:45:57.834413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.834048 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-envoy\") pod \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " Oct 29 23:45:57.834413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.834123 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-workload-socket\") pod \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " Oct 29 23:45:57.834413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.834177 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-kube-api-access-st2d6\" (UniqueName: \"kubernetes.io/projected/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-kube-api-access-st2d6\") pod \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " Oct 29 23:45:57.834413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.834251 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-podinfo\") pod \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " Oct 29 23:45:57.834805 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.834538 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-token\" (UniqueName: \"kubernetes.io/projected/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-token\") pod \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " Oct 29 23:45:57.834805 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.834692 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-data\") pod \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " Oct 29 23:45:57.834805 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.834757 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-workload-certs\") pod \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " Oct 29 23:45:57.834805 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.834801 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-credential-socket\") pod \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " Oct 29 23:45:57.835017 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.834829 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istiod-ca-cert\") pod \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\" (UID: \"7dfdb32d-7346-42ac-a5e6-3081f0895a8e\") " Oct 29 23:45:57.836220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.835980 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istiod-ca-cert" pod "7dfdb32d-7346-42ac-a5e6-3081f0895a8e" (UID: "7dfdb32d-7346-42ac-a5e6-3081f0895a8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:57.837252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.836855 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-workload-certs" pod "7dfdb32d-7346-42ac-a5e6-3081f0895a8e" (UID: "7dfdb32d-7346-42ac-a5e6-3081f0895a8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:57.837252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.837044 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-data" pod "7dfdb32d-7346-42ac-a5e6-3081f0895a8e" (UID: "7dfdb32d-7346-42ac-a5e6-3081f0895a8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:57.837252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.837236 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-credential-socket" pod "7dfdb32d-7346-42ac-a5e6-3081f0895a8e" (UID: "7dfdb32d-7346-42ac-a5e6-3081f0895a8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:57.838367 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.838048 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-workload-socket" pod "7dfdb32d-7346-42ac-a5e6-3081f0895a8e" (UID: "7dfdb32d-7346-42ac-a5e6-3081f0895a8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:57.841518 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.840130 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-kube-api-access-st2d6" pod "7dfdb32d-7346-42ac-a5e6-3081f0895a8e" (UID: "7dfdb32d-7346-42ac-a5e6-3081f0895a8e"). InnerVolumeSpecName "kube-api-access-st2d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:57.844647 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.844613 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-token" pod "7dfdb32d-7346-42ac-a5e6-3081f0895a8e" (UID: "7dfdb32d-7346-42ac-a5e6-3081f0895a8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:57.845648 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.845616 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-envoy" pod "7dfdb32d-7346-42ac-a5e6-3081f0895a8e" (UID: "7dfdb32d-7346-42ac-a5e6-3081f0895a8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:57.846257 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.846231 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-podinfo" pod "7dfdb32d-7346-42ac-a5e6-3081f0895a8e" (UID: "7dfdb32d-7346-42ac-a5e6-3081f0895a8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:45:57.980269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.977342 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:57.980269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.977380 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:57.980269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.977395 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:57.980269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.977408 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:57.980269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.977421 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st2d6\" (UniqueName: \"kubernetes.io/projected/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-kube-api-access-st2d6\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:57.980269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.977436 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:57.980269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.977450 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:57.980269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.977461 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:57.980269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:57.977474 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dfdb32d-7346-42ac-a5e6-3081f0895a8e-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:45:58.112325 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:58.112293 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg"] Oct 29 23:45:58.135339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:58.135302 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-fp5cg"] Oct 29 23:45:59.554651 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:45:59.554615 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dfdb32d-7346-42ac-a5e6-3081f0895a8e" path="/var/lib/kubelet/pods/7dfdb32d-7346-42ac-a5e6-3081f0895a8e/volumes" Oct 29 23:46:03.998131 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:46:03.996009 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:46:13.925540 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:13.925503 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw"] Oct 29 23:46:13.926492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:13.926474 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b7893cca-8dc6-4395-bc25-452bcd512dcd" containerName="istio-proxy" Oct 29 23:46:13.926492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:13.926492 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7893cca-8dc6-4395-bc25-452bcd512dcd" containerName="istio-proxy" Oct 29 23:46:13.926639 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:13.926518 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b783e0d4-8272-4c3e-b046-0d6a70160fea" containerName="istio-proxy" Oct 29 23:46:13.926639 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:13.926525 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b783e0d4-8272-4c3e-b046-0d6a70160fea" containerName="istio-proxy" Oct 29 23:46:13.926639 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:13.926541 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7dfdb32d-7346-42ac-a5e6-3081f0895a8e" containerName="istio-proxy" Oct 29 23:46:13.926639 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:13.926546 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dfdb32d-7346-42ac-a5e6-3081f0895a8e" containerName="istio-proxy" Oct 29 23:46:13.926639 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:13.926561 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b783e0d4-8272-4c3e-b046-0d6a70160fea" containerName="istio-proxy" Oct 29 23:46:13.926639 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:13.926566 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b783e0d4-8272-4c3e-b046-0d6a70160fea" containerName="istio-proxy" Oct 29 23:46:13.926639 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:13.926586 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7dfdb32d-7346-42ac-a5e6-3081f0895a8e" containerName="istio-proxy" Oct 29 23:46:13.926639 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:13.926591 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dfdb32d-7346-42ac-a5e6-3081f0895a8e" containerName="istio-proxy" Oct 29 23:46:13.926908 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:13.926837 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b783e0d4-8272-4c3e-b046-0d6a70160fea" containerName="istio-proxy" Oct 29 23:46:13.926908 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:13.926869 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dfdb32d-7346-42ac-a5e6-3081f0895a8e" containerName="istio-proxy" Oct 29 23:46:13.926908 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:13.926886 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b783e0d4-8272-4c3e-b046-0d6a70160fea" containerName="istio-proxy" Oct 29 23:46:13.926908 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:13.926896 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dfdb32d-7346-42ac-a5e6-3081f0895a8e" containerName="istio-proxy" Oct 29 23:46:13.927036 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:13.926922 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7893cca-8dc6-4395-bc25-452bcd512dcd" containerName="istio-proxy" Oct 29 23:46:13.927036 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:13.926936 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7893cca-8dc6-4395-bc25-452bcd512dcd" containerName="istio-proxy" Oct 29 23:46:13.931678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:13.931659 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:13.974241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:13.969739 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw"] Oct 29 23:46:14.112389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.112346 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.112590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.112462 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.112590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.112490 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ghvk\" (UniqueName: \"kubernetes.io/projected/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-kube-api-access-9ghvk\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.112590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.112559 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.112738 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.112717 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.112787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.112764 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.112878 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.112865 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.112916 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.112891 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.113111 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.113066 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.214437 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.214348 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.214636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.214455 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.214636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.214486 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.214768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.214649 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.214768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.214686 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.214768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.214747 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.214967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.214775 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ghvk\" (UniqueName: \"kubernetes.io/projected/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-kube-api-access-9ghvk\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.214967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.214807 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.214967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.214816 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.214967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.214883 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.215280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.215045 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.215280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.215071 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.215280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.215174 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.215548 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.215522 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.217637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.217617 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.217753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.217696 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.242455 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.242422 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ghvk\" (UniqueName: \"kubernetes.io/projected/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-kube-api-access-9ghvk\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.242455 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.242451 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.252061 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.252028 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:14.583201 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:46:14.583140 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf37ad5_2491_488a_9cea_5f0a3872f5fb.slice/crio-74416197d46fd1d40abdccfff2a98e5f6d1eefc0a1466abaecfd3dfc0becd8b0 WatchSource:0}: Error finding container 74416197d46fd1d40abdccfff2a98e5f6d1eefc0a1466abaecfd3dfc0becd8b0: Status 404 returned error can't find the container with id 74416197d46fd1d40abdccfff2a98e5f6d1eefc0a1466abaecfd3dfc0becd8b0 Oct 29 23:46:14.583521 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.583494 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw"] Oct 29 23:46:14.587293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.587251 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:46:14.587395 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.587343 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:46:14.587448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.587394 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:46:14.925936 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.925898 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" event={"ID":"2bf37ad5-2491-488a-9cea-5f0a3872f5fb","Type":"ContainerStarted","Data":"48261d81e8d198bc853f0369345494e3611da5dd69f97a79e36e0637ac1fb174"} Oct 29 23:46:14.926452 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.925966 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" event={"ID":"2bf37ad5-2491-488a-9cea-5f0a3872f5fb","Type":"ContainerStarted","Data":"74416197d46fd1d40abdccfff2a98e5f6d1eefc0a1466abaecfd3dfc0becd8b0"} Oct 29 23:46:14.984871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:14.984802 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" podStartSLOduration=1.984784001 podStartE2EDuration="1.984784001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:46:14.983355575 +0000 UTC m=+3821.634854115" watchObservedRunningTime="2026-10-29 23:46:14.984784001 +0000 UTC m=+3821.636282531" Oct 29 23:46:15.252922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:15.252834 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:15.254369 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:15.254345 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Oct 29 23:46:15.254500 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:15.254403 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" podUID="2bf37ad5-2491-488a-9cea-5f0a3872f5fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Oct 29 23:46:16.254750 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:16.254653 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Oct 29 23:46:16.256028 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:16.255990 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" podUID="2bf37ad5-2491-488a-9cea-5f0a3872f5fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Oct 29 23:46:17.253437 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:17.252820 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Oct 29 23:46:17.253437 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:17.252897 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" podUID="2bf37ad5-2491-488a-9cea-5f0a3872f5fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Oct 29 23:46:18.253612 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:18.253573 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Oct 29 23:46:18.254222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:18.254169 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" podUID="2bf37ad5-2491-488a-9cea-5f0a3872f5fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Oct 29 23:46:19.258568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:19.258529 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Oct 29 23:46:19.259001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:19.258589 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" podUID="2bf37ad5-2491-488a-9cea-5f0a3872f5fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Oct 29 23:46:20.253210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:20.252707 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Oct 29 23:46:20.253210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:20.252800 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" podUID="2bf37ad5-2491-488a-9cea-5f0a3872f5fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Oct 29 23:46:21.259341 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:21.259300 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:21.260368 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:21.260328 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:21.261308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:21.261280 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:46:24.068005 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:24.067970 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:46:28.976615 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:28.976580 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j"] Oct 29 23:46:28.977532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:28.977516 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b7893cca-8dc6-4395-bc25-452bcd512dcd" containerName="istio-proxy" Oct 29 23:46:28.977591 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:28.977534 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7893cca-8dc6-4395-bc25-452bcd512dcd" containerName="istio-proxy" Oct 29 23:46:28.982942 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:28.982920 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.009570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.009527 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j"] Oct 29 23:46:29.109520 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.109479 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.109743 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.109620 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb5cr\" (UniqueName: \"kubernetes.io/projected/f97576f7-0dd7-4515-9774-8e8849cc2c5e-kube-api-access-xb5cr\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.109743 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.109735 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.109843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.109771 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.109954 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.109931 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.110068 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.110045 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.110239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.110220 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.110335 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.110312 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.110399 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.110385 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.211434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.211401 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.211646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.211493 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.211646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.211624 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.211790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.211674 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.211790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.211719 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.211790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.211765 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.211960 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.211802 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb5cr\" (UniqueName: \"kubernetes.io/projected/f97576f7-0dd7-4515-9774-8e8849cc2c5e-kube-api-access-xb5cr\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.211960 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.211853 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.211960 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.211876 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.212083 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.211964 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.212261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.212242 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.212335 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.212286 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.212551 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.212530 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.212595 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.212562 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.214076 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.214049 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.214762 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.214734 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.236827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.236762 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.237289 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.237262 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb5cr\" (UniqueName: \"kubernetes.io/projected/f97576f7-0dd7-4515-9774-8e8849cc2c5e-kube-api-access-xb5cr\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.303844 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.303807 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:29.642925 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:46:29.642885 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97576f7_0dd7_4515_9774_8e8849cc2c5e.slice/crio-0b11b60ea71bcdbdd7fd17afbcaa9cd98b6856c30c8be75797c5b94dc0761e6d WatchSource:0}: Error finding container 0b11b60ea71bcdbdd7fd17afbcaa9cd98b6856c30c8be75797c5b94dc0761e6d: Status 404 returned error can't find the container with id 0b11b60ea71bcdbdd7fd17afbcaa9cd98b6856c30c8be75797c5b94dc0761e6d Oct 29 23:46:29.644646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.644615 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j"] Oct 29 23:46:29.647494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.647457 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:46:29.647619 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.647537 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:46:29.647619 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:29.647570 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:46:30.512037 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:30.511998 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" event={"ID":"f97576f7-0dd7-4515-9774-8e8849cc2c5e","Type":"ContainerStarted","Data":"e84bfaeb37855ca31279d661e919c38f573366c6cb038f3f0577484fda345a24"} Oct 29 23:46:30.512493 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:30.512063 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" event={"ID":"f97576f7-0dd7-4515-9774-8e8849cc2c5e","Type":"ContainerStarted","Data":"0b11b60ea71bcdbdd7fd17afbcaa9cd98b6856c30c8be75797c5b94dc0761e6d"} Oct 29 23:46:30.586423 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:30.586367 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" podStartSLOduration=2.586348115 podStartE2EDuration="2.586348115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:46:30.584596262 +0000 UTC m=+3837.236094792" watchObservedRunningTime="2026-10-29 23:46:30.586348115 +0000 UTC m=+3837.237846644" Oct 29 23:46:31.304581 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:31.304547 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:31.306171 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:31.306105 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body= Oct 29 23:46:31.306171 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:31.306156 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" podUID="f97576f7-0dd7-4515-9774-8e8849cc2c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" Oct 29 23:46:32.317212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:32.312291 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body= Oct 29 23:46:32.317212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:32.312358 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" podUID="f97576f7-0dd7-4515-9774-8e8849cc2c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" Oct 29 23:46:33.305052 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:33.305014 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body= Oct 29 23:46:33.305289 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:33.305080 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" podUID="f97576f7-0dd7-4515-9774-8e8849cc2c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" Oct 29 23:46:33.928210 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:46:33.921740 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:46:34.304846 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:34.304770 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body= Oct 29 23:46:34.304846 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:34.304820 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" podUID="f97576f7-0dd7-4515-9774-8e8849cc2c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" Oct 29 23:46:35.304385 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:35.304350 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body= Oct 29 23:46:35.304886 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:35.304404 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" podUID="f97576f7-0dd7-4515-9774-8e8849cc2c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" Oct 29 23:46:36.373447 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:36.372790 5057 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body= Oct 29 23:46:36.374708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:36.374674 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" podUID="f97576f7-0dd7-4515-9774-8e8849cc2c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" Oct 29 23:46:37.099347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:37.099312 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s"] Oct 29 23:46:37.099623 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:37.099569 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" podUID="1729a889-e3d1-43df-89d1-f3356bc34a46" containerName="istio-proxy" containerID="cri-o://2df8e5154fbbae8711f860b55eb2af6e5b9efed71a987260cdaf8acbccc3f755" gracePeriod=30 Oct 29 23:46:37.642629 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:37.642595 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:37.643593 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:37.643566 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:37.644756 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:37.644737 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:46:40.241349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:40.241314 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8"] Oct 29 23:46:40.241830 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:40.241625 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" podUID="271ab84e-ada3-4de0-8cbb-b25424172421" containerName="istio-proxy" containerID="cri-o://8a5ff8d861ee371380177a01d33089189e85befb1d2dbd8b7e0a21a8f9fe26e4" gracePeriod=30 Oct 29 23:46:42.432639 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.432605 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:46:42.553557 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.553525 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-envoy\") pod \"1729a889-e3d1-43df-89d1-f3356bc34a46\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " Oct 29 23:46:42.553747 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.553582 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-workload-socket\") pod \"1729a889-e3d1-43df-89d1-f3356bc34a46\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " Oct 29 23:46:42.553747 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.553617 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-credential-socket\") pod \"1729a889-e3d1-43df-89d1-f3356bc34a46\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " Oct 29 23:46:42.553747 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.553732 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-workload-certs\") pod \"1729a889-e3d1-43df-89d1-f3356bc34a46\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " Oct 29 23:46:42.553963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.553891 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-workload-socket" pod "1729a889-e3d1-43df-89d1-f3356bc34a46" (UID: "1729a889-e3d1-43df-89d1-f3356bc34a46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:42.553963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.553955 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1729a889-e3d1-43df-89d1-f3356bc34a46-kube-api-access-ht76t\" (UniqueName: \"kubernetes.io/projected/1729a889-e3d1-43df-89d1-f3356bc34a46-kube-api-access-ht76t\") pod \"1729a889-e3d1-43df-89d1-f3356bc34a46\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " Oct 29 23:46:42.554067 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.554034 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-token\" (UniqueName: \"kubernetes.io/projected/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-token\") pod \"1729a889-e3d1-43df-89d1-f3356bc34a46\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " Oct 29 23:46:42.554124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.554079 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-podinfo\") pod \"1729a889-e3d1-43df-89d1-f3356bc34a46\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " Oct 29 23:46:42.554249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.554225 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1729a889-e3d1-43df-89d1-f3356bc34a46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1729a889-e3d1-43df-89d1-f3356bc34a46-istiod-ca-cert\") pod \"1729a889-e3d1-43df-89d1-f3356bc34a46\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " Oct 29 23:46:42.554400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.554272 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-data\") pod \"1729a889-e3d1-43df-89d1-f3356bc34a46\" (UID: \"1729a889-e3d1-43df-89d1-f3356bc34a46\") " Oct 29 23:46:42.555250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.555226 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:46:42.555438 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.555390 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-credential-socket" pod "1729a889-e3d1-43df-89d1-f3356bc34a46" (UID: "1729a889-e3d1-43df-89d1-f3356bc34a46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:42.555521 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.555464 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-workload-certs" pod "1729a889-e3d1-43df-89d1-f3356bc34a46" (UID: "1729a889-e3d1-43df-89d1-f3356bc34a46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:42.555743 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.555718 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-data" pod "1729a889-e3d1-43df-89d1-f3356bc34a46" (UID: "1729a889-e3d1-43df-89d1-f3356bc34a46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:42.555839 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.555820 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1729a889-e3d1-43df-89d1-f3356bc34a46-istiod-ca-cert" pod "1729a889-e3d1-43df-89d1-f3356bc34a46" (UID: "1729a889-e3d1-43df-89d1-f3356bc34a46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:46:42.556631 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.556597 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-envoy" pod "1729a889-e3d1-43df-89d1-f3356bc34a46" (UID: "1729a889-e3d1-43df-89d1-f3356bc34a46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:42.557996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.557977 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1729a889-e3d1-43df-89d1-f3356bc34a46-kube-api-access-ht76t" pod "1729a889-e3d1-43df-89d1-f3356bc34a46" (UID: "1729a889-e3d1-43df-89d1-f3356bc34a46"). InnerVolumeSpecName "kube-api-access-ht76t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:46:42.558096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.558005 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-token" pod "1729a889-e3d1-43df-89d1-f3356bc34a46" (UID: "1729a889-e3d1-43df-89d1-f3356bc34a46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:46:42.558259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.558234 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-podinfo" pod "1729a889-e3d1-43df-89d1-f3356bc34a46" (UID: "1729a889-e3d1-43df-89d1-f3356bc34a46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:46:42.662260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.661968 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1729a889-e3d1-43df-89d1-f3356bc34a46-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:46:42.662260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.662011 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:46:42.662260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.662025 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:46:42.662260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.662038 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:46:42.662260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.662053 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1729a889-e3d1-43df-89d1-f3356bc34a46-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:46:42.662260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.662066 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht76t\" (UniqueName: \"kubernetes.io/projected/1729a889-e3d1-43df-89d1-f3356bc34a46-kube-api-access-ht76t\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:46:42.662260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.662080 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:46:42.662260 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.662093 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1729a889-e3d1-43df-89d1-f3356bc34a46-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:46:42.942960 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.942924 5057 generic.go:356] "Generic (PLEG): container finished" podID="1729a889-e3d1-43df-89d1-f3356bc34a46" containerID="2df8e5154fbbae8711f860b55eb2af6e5b9efed71a987260cdaf8acbccc3f755" exitCode=0 Oct 29 23:46:42.943181 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.942989 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" Oct 29 23:46:42.943181 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.943001 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" event={"ID":"1729a889-e3d1-43df-89d1-f3356bc34a46","Type":"ContainerDied","Data":"2df8e5154fbbae8711f860b55eb2af6e5b9efed71a987260cdaf8acbccc3f755"} Oct 29 23:46:42.943181 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.943071 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s" event={"ID":"1729a889-e3d1-43df-89d1-f3356bc34a46","Type":"ContainerDied","Data":"a33bbc4fcf510252f811f4e65235b1d510f4bf19b955187d82b211f861287429"} Oct 29 23:46:42.943181 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.943106 5057 scope.go:122] "RemoveContainer" containerID="2df8e5154fbbae8711f860b55eb2af6e5b9efed71a987260cdaf8acbccc3f755" Oct 29 23:46:42.957315 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.957290 5057 scope.go:122] "RemoveContainer" containerID="233a3bc3f3ec7b97f979dc0789e6e4b2cf6c87a2b201fcec7d6967f4e0bc98c5" Oct 29 23:46:42.968576 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.968556 5057 scope.go:122] "RemoveContainer" containerID="2df8e5154fbbae8711f860b55eb2af6e5b9efed71a987260cdaf8acbccc3f755" Oct 29 23:46:42.968905 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:46:42.968882 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2df8e5154fbbae8711f860b55eb2af6e5b9efed71a987260cdaf8acbccc3f755\": container with ID starting with 2df8e5154fbbae8711f860b55eb2af6e5b9efed71a987260cdaf8acbccc3f755 not found: ID does not exist" containerID="2df8e5154fbbae8711f860b55eb2af6e5b9efed71a987260cdaf8acbccc3f755" Oct 29 23:46:42.969021 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.968915 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df8e5154fbbae8711f860b55eb2af6e5b9efed71a987260cdaf8acbccc3f755"} err="failed to get container status \"2df8e5154fbbae8711f860b55eb2af6e5b9efed71a987260cdaf8acbccc3f755\": rpc error: code = NotFound desc = could not find container \"2df8e5154fbbae8711f860b55eb2af6e5b9efed71a987260cdaf8acbccc3f755\": container with ID starting with 2df8e5154fbbae8711f860b55eb2af6e5b9efed71a987260cdaf8acbccc3f755 not found: ID does not exist" Oct 29 23:46:42.969021 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.968944 5057 scope.go:122] "RemoveContainer" containerID="233a3bc3f3ec7b97f979dc0789e6e4b2cf6c87a2b201fcec7d6967f4e0bc98c5" Oct 29 23:46:42.969267 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:46:42.969242 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"233a3bc3f3ec7b97f979dc0789e6e4b2cf6c87a2b201fcec7d6967f4e0bc98c5\": container with ID starting with 233a3bc3f3ec7b97f979dc0789e6e4b2cf6c87a2b201fcec7d6967f4e0bc98c5 not found: ID does not exist" containerID="233a3bc3f3ec7b97f979dc0789e6e4b2cf6c87a2b201fcec7d6967f4e0bc98c5" Oct 29 23:46:42.969336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.969278 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233a3bc3f3ec7b97f979dc0789e6e4b2cf6c87a2b201fcec7d6967f4e0bc98c5"} err="failed to get container status \"233a3bc3f3ec7b97f979dc0789e6e4b2cf6c87a2b201fcec7d6967f4e0bc98c5\": rpc error: code = NotFound desc = could not find container \"233a3bc3f3ec7b97f979dc0789e6e4b2cf6c87a2b201fcec7d6967f4e0bc98c5\": container with ID starting with 233a3bc3f3ec7b97f979dc0789e6e4b2cf6c87a2b201fcec7d6967f4e0bc98c5 not found: ID does not exist" Oct 29 23:46:42.998492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:42.998464 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s"] Oct 29 23:46:43.010667 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:43.010642 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-trd5s"] Oct 29 23:46:43.485117 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:43.485081 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1729a889-e3d1-43df-89d1-f3356bc34a46" path="/var/lib/kubelet/pods/1729a889-e3d1-43df-89d1-f3356bc34a46/volumes" Oct 29 23:46:45.570112 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.570085 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:46:45.618530 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.618496 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/271ab84e-ada3-4de0-8cbb-b25424172421-kube-api-access-dcspz\" (UniqueName: \"kubernetes.io/projected/271ab84e-ada3-4de0-8cbb-b25424172421-kube-api-access-dcspz\") pod \"271ab84e-ada3-4de0-8cbb-b25424172421\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " Oct 29 23:46:45.618722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.618674 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-istio-data\") pod \"271ab84e-ada3-4de0-8cbb-b25424172421\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " Oct 29 23:46:45.618802 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.618761 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-workload-socket\") pod \"271ab84e-ada3-4de0-8cbb-b25424172421\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " Oct 29 23:46:45.618871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.618824 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-credential-socket\") pod \"271ab84e-ada3-4de0-8cbb-b25424172421\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " Oct 29 23:46:45.618931 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.618877 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-workload-certs\") pod \"271ab84e-ada3-4de0-8cbb-b25424172421\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " Oct 29 23:46:45.618931 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.618918 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/271ab84e-ada3-4de0-8cbb-b25424172421-istio-token\" (UniqueName: \"kubernetes.io/projected/271ab84e-ada3-4de0-8cbb-b25424172421-istio-token\") pod \"271ab84e-ada3-4de0-8cbb-b25424172421\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " Oct 29 23:46:45.619038 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.619006 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-istio-data" pod "271ab84e-ada3-4de0-8cbb-b25424172421" (UID: "271ab84e-ada3-4de0-8cbb-b25424172421"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:45.619038 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.619013 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/271ab84e-ada3-4de0-8cbb-b25424172421-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/271ab84e-ada3-4de0-8cbb-b25424172421-istio-podinfo\") pod \"271ab84e-ada3-4de0-8cbb-b25424172421\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " Oct 29 23:46:45.619159 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.619030 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-workload-socket" pod "271ab84e-ada3-4de0-8cbb-b25424172421" (UID: "271ab84e-ada3-4de0-8cbb-b25424172421"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:45.619159 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.619124 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-istio-envoy\") pod \"271ab84e-ada3-4de0-8cbb-b25424172421\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " Oct 29 23:46:45.619159 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.619134 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-credential-socket" pod "271ab84e-ada3-4de0-8cbb-b25424172421" (UID: "271ab84e-ada3-4de0-8cbb-b25424172421"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:45.619393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.619170 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-workload-certs" pod "271ab84e-ada3-4de0-8cbb-b25424172421" (UID: "271ab84e-ada3-4de0-8cbb-b25424172421"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:45.619393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.619261 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/271ab84e-ada3-4de0-8cbb-b25424172421-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/271ab84e-ada3-4de0-8cbb-b25424172421-istiod-ca-cert\") pod \"271ab84e-ada3-4de0-8cbb-b25424172421\" (UID: \"271ab84e-ada3-4de0-8cbb-b25424172421\") " Oct 29 23:46:45.619592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.619566 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271ab84e-ada3-4de0-8cbb-b25424172421-istiod-ca-cert" pod "271ab84e-ada3-4de0-8cbb-b25424172421" (UID: "271ab84e-ada3-4de0-8cbb-b25424172421"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:46:45.619964 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.619946 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:46:45.620061 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.619966 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:46:45.620061 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.619975 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:46:45.620061 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.619987 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:46:45.620061 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.619999 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/271ab84e-ada3-4de0-8cbb-b25424172421-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:46:45.620985 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.620963 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271ab84e-ada3-4de0-8cbb-b25424172421-kube-api-access-dcspz" pod "271ab84e-ada3-4de0-8cbb-b25424172421" (UID: "271ab84e-ada3-4de0-8cbb-b25424172421"). InnerVolumeSpecName "kube-api-access-dcspz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:46:45.621113 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.621095 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271ab84e-ada3-4de0-8cbb-b25424172421-istio-token" pod "271ab84e-ada3-4de0-8cbb-b25424172421" (UID: "271ab84e-ada3-4de0-8cbb-b25424172421"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:46:45.621628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.621602 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/271ab84e-ada3-4de0-8cbb-b25424172421-istio-podinfo" pod "271ab84e-ada3-4de0-8cbb-b25424172421" (UID: "271ab84e-ada3-4de0-8cbb-b25424172421"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:46:45.621721 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.621606 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-istio-envoy" pod "271ab84e-ada3-4de0-8cbb-b25424172421" (UID: "271ab84e-ada3-4de0-8cbb-b25424172421"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:45.721684 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.721645 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/271ab84e-ada3-4de0-8cbb-b25424172421-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:46:45.721684 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.721678 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/271ab84e-ada3-4de0-8cbb-b25424172421-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:46:45.721684 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.721687 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcspz\" (UniqueName: \"kubernetes.io/projected/271ab84e-ada3-4de0-8cbb-b25424172421-kube-api-access-dcspz\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:46:45.721966 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.721697 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/271ab84e-ada3-4de0-8cbb-b25424172421-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:46:45.988122 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.988089 5057 generic.go:356] "Generic (PLEG): container finished" podID="271ab84e-ada3-4de0-8cbb-b25424172421" containerID="8a5ff8d861ee371380177a01d33089189e85befb1d2dbd8b7e0a21a8f9fe26e4" exitCode=0 Oct 29 23:46:45.988355 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.988140 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" event={"ID":"271ab84e-ada3-4de0-8cbb-b25424172421","Type":"ContainerDied","Data":"8a5ff8d861ee371380177a01d33089189e85befb1d2dbd8b7e0a21a8f9fe26e4"} Oct 29 23:46:45.988355 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.988179 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" event={"ID":"271ab84e-ada3-4de0-8cbb-b25424172421","Type":"ContainerDied","Data":"c8203a73658e27b314d43ce7bd5e6a83b95b5c81abecb85eebcbf95f18fb8edc"} Oct 29 23:46:45.988355 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.988142 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8" Oct 29 23:46:45.988355 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:45.988224 5057 scope.go:122] "RemoveContainer" containerID="8a5ff8d861ee371380177a01d33089189e85befb1d2dbd8b7e0a21a8f9fe26e4" Oct 29 23:46:46.001095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:46.001075 5057 scope.go:122] "RemoveContainer" containerID="77b1cb97d7cc92f7183a56aca462767ede0342c308d7ae2382514b54e6e52bf1" Oct 29 23:46:46.012125 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:46.012104 5057 scope.go:122] "RemoveContainer" containerID="8a5ff8d861ee371380177a01d33089189e85befb1d2dbd8b7e0a21a8f9fe26e4" Oct 29 23:46:46.012452 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:46:46.012428 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a5ff8d861ee371380177a01d33089189e85befb1d2dbd8b7e0a21a8f9fe26e4\": container with ID starting with 8a5ff8d861ee371380177a01d33089189e85befb1d2dbd8b7e0a21a8f9fe26e4 not found: ID does not exist" containerID="8a5ff8d861ee371380177a01d33089189e85befb1d2dbd8b7e0a21a8f9fe26e4" Oct 29 23:46:46.012552 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:46.012462 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a5ff8d861ee371380177a01d33089189e85befb1d2dbd8b7e0a21a8f9fe26e4"} err="failed to get container status \"8a5ff8d861ee371380177a01d33089189e85befb1d2dbd8b7e0a21a8f9fe26e4\": rpc error: code = NotFound desc = could not find container \"8a5ff8d861ee371380177a01d33089189e85befb1d2dbd8b7e0a21a8f9fe26e4\": container with ID starting with 8a5ff8d861ee371380177a01d33089189e85befb1d2dbd8b7e0a21a8f9fe26e4 not found: ID does not exist" Oct 29 23:46:46.012552 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:46.012486 5057 scope.go:122] "RemoveContainer" containerID="77b1cb97d7cc92f7183a56aca462767ede0342c308d7ae2382514b54e6e52bf1" Oct 29 23:46:46.012775 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:46:46.012754 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77b1cb97d7cc92f7183a56aca462767ede0342c308d7ae2382514b54e6e52bf1\": container with ID starting with 77b1cb97d7cc92f7183a56aca462767ede0342c308d7ae2382514b54e6e52bf1 not found: ID does not exist" containerID="77b1cb97d7cc92f7183a56aca462767ede0342c308d7ae2382514b54e6e52bf1" Oct 29 23:46:46.012833 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:46.012779 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77b1cb97d7cc92f7183a56aca462767ede0342c308d7ae2382514b54e6e52bf1"} err="failed to get container status \"77b1cb97d7cc92f7183a56aca462767ede0342c308d7ae2382514b54e6e52bf1\": rpc error: code = NotFound desc = could not find container \"77b1cb97d7cc92f7183a56aca462767ede0342c308d7ae2382514b54e6e52bf1\": container with ID starting with 77b1cb97d7cc92f7183a56aca462767ede0342c308d7ae2382514b54e6e52bf1 not found: ID does not exist" Oct 29 23:46:46.045939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:46.045906 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8"] Oct 29 23:46:46.061206 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:46.058304 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-6f162a-openshift-default-77bd944489-p7hh8"] Oct 29 23:46:47.484731 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:47.484698 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="271ab84e-ada3-4de0-8cbb-b25424172421" path="/var/lib/kubelet/pods/271ab84e-ada3-4de0-8cbb-b25424172421/volumes" Oct 29 23:46:48.121110 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:48.121075 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt"] Oct 29 23:46:53.121665 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:53.121624 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt" podUID="f2c68413-7dbd-4108-ad58-37599bbe9f8e" containerName="payload-processing" containerID="cri-o://b29f7735ef6899bbd393f193edebcdf14a0f33ae9ce763bfe95b727a334da0be" gracePeriod=25 Oct 29 23:46:53.436418 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:53.436386 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt" Oct 29 23:46:53.550141 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:53.550109 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c68413-7dbd-4108-ad58-37599bbe9f8e-plugins-config-volume" pod "f2c68413-7dbd-4108-ad58-37599bbe9f8e" (UID: "f2c68413-7dbd-4108-ad58-37599bbe9f8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:46:53.551739 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:53.551698 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c68413-7dbd-4108-ad58-37599bbe9f8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c68413-7dbd-4108-ad58-37599bbe9f8e-plugins-config-volume\") pod \"f2c68413-7dbd-4108-ad58-37599bbe9f8e\" (UID: \"f2c68413-7dbd-4108-ad58-37599bbe9f8e\") " Oct 29 23:46:53.558592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:53.558539 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c68413-7dbd-4108-ad58-37599bbe9f8e-kube-api-access-xm6s9\" (UniqueName: \"kubernetes.io/projected/f2c68413-7dbd-4108-ad58-37599bbe9f8e-kube-api-access-xm6s9\") pod \"f2c68413-7dbd-4108-ad58-37599bbe9f8e\" (UID: \"f2c68413-7dbd-4108-ad58-37599bbe9f8e\") " Oct 29 23:46:53.559571 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:53.559544 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c68413-7dbd-4108-ad58-37599bbe9f8e-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:46:53.563769 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:53.563746 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c68413-7dbd-4108-ad58-37599bbe9f8e-kube-api-access-xm6s9" pod "f2c68413-7dbd-4108-ad58-37599bbe9f8e" (UID: "f2c68413-7dbd-4108-ad58-37599bbe9f8e"). InnerVolumeSpecName "kube-api-access-xm6s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:46:53.660954 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:53.660869 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm6s9\" (UniqueName: \"kubernetes.io/projected/f2c68413-7dbd-4108-ad58-37599bbe9f8e-kube-api-access-xm6s9\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:46:54.175217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:54.171931 5057 generic.go:356] "Generic (PLEG): container finished" podID="f2c68413-7dbd-4108-ad58-37599bbe9f8e" containerID="b29f7735ef6899bbd393f193edebcdf14a0f33ae9ce763bfe95b727a334da0be" exitCode=0 Oct 29 23:46:54.175217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:54.172363 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt" event={"ID":"f2c68413-7dbd-4108-ad58-37599bbe9f8e","Type":"ContainerDied","Data":"b29f7735ef6899bbd393f193edebcdf14a0f33ae9ce763bfe95b727a334da0be"} Oct 29 23:46:54.186208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:54.185353 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt" Oct 29 23:46:54.198209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:54.191354 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt" event={"ID":"f2c68413-7dbd-4108-ad58-37599bbe9f8e","Type":"ContainerDied","Data":"19aa93517f4c3db87039261b5a793ef934bad615c85f01a427e58715162c3ca4"} Oct 29 23:46:54.198209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:54.191451 5057 scope.go:122] "RemoveContainer" containerID="b29f7735ef6899bbd393f193edebcdf14a0f33ae9ce763bfe95b727a334da0be" Oct 29 23:46:54.278810 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:54.276320 5057 scope.go:122] "RemoveContainer" containerID="b29f7735ef6899bbd393f193edebcdf14a0f33ae9ce763bfe95b727a334da0be" Oct 29 23:46:54.286208 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:46:54.280638 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b29f7735ef6899bbd393f193edebcdf14a0f33ae9ce763bfe95b727a334da0be\": container with ID starting with b29f7735ef6899bbd393f193edebcdf14a0f33ae9ce763bfe95b727a334da0be not found: ID does not exist" containerID="b29f7735ef6899bbd393f193edebcdf14a0f33ae9ce763bfe95b727a334da0be" Oct 29 23:46:54.286208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:54.280752 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b29f7735ef6899bbd393f193edebcdf14a0f33ae9ce763bfe95b727a334da0be"} err="failed to get container status \"b29f7735ef6899bbd393f193edebcdf14a0f33ae9ce763bfe95b727a334da0be\": rpc error: code = NotFound desc = could not find container \"b29f7735ef6899bbd393f193edebcdf14a0f33ae9ce763bfe95b727a334da0be\": container with ID starting with b29f7735ef6899bbd393f193edebcdf14a0f33ae9ce763bfe95b727a334da0be not found: ID does not exist" Oct 29 23:46:54.361520 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:54.361465 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt"] Oct 29 23:46:54.367545 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:54.367507 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b7a567-867cf68898-n5zjt"] Oct 29 23:46:55.485533 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.485501 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c68413-7dbd-4108-ad58-37599bbe9f8e" path="/var/lib/kubelet/pods/f2c68413-7dbd-4108-ad58-37599bbe9f8e/volumes" Oct 29 23:46:55.551342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.551308 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6"] Oct 29 23:46:55.552220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.552181 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1729a889-e3d1-43df-89d1-f3356bc34a46" containerName="istio-proxy" Oct 29 23:46:55.552220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.552216 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1729a889-e3d1-43df-89d1-f3356bc34a46" containerName="istio-proxy" Oct 29 23:46:55.552389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.552228 5057 cpu_manager.go:414] "containerMap: removing container" podUID="271ab84e-ada3-4de0-8cbb-b25424172421" containerName="istio-proxy" Oct 29 23:46:55.552389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.552233 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="271ab84e-ada3-4de0-8cbb-b25424172421" containerName="istio-proxy" Oct 29 23:46:55.552389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.552247 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f2c68413-7dbd-4108-ad58-37599bbe9f8e" containerName="payload-processing" Oct 29 23:46:55.552389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.552252 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c68413-7dbd-4108-ad58-37599bbe9f8e" containerName="payload-processing" Oct 29 23:46:55.552389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.552262 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1729a889-e3d1-43df-89d1-f3356bc34a46" containerName="istio-proxy" Oct 29 23:46:55.552389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.552267 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1729a889-e3d1-43df-89d1-f3356bc34a46" containerName="istio-proxy" Oct 29 23:46:55.552389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.552288 5057 cpu_manager.go:414] "containerMap: removing container" podUID="271ab84e-ada3-4de0-8cbb-b25424172421" containerName="istio-proxy" Oct 29 23:46:55.552389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.552292 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="271ab84e-ada3-4de0-8cbb-b25424172421" containerName="istio-proxy" Oct 29 23:46:55.552653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.552433 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1729a889-e3d1-43df-89d1-f3356bc34a46" containerName="istio-proxy" Oct 29 23:46:55.552653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.552441 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="271ab84e-ada3-4de0-8cbb-b25424172421" containerName="istio-proxy" Oct 29 23:46:55.552653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.552449 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c68413-7dbd-4108-ad58-37599bbe9f8e" containerName="payload-processing" Oct 29 23:46:55.552653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.552458 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1729a889-e3d1-43df-89d1-f3356bc34a46" containerName="istio-proxy" Oct 29 23:46:55.552653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.552466 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="271ab84e-ada3-4de0-8cbb-b25424172421" containerName="istio-proxy" Oct 29 23:46:55.557962 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.557940 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.574724 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.574700 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3b8149-openshift-default-dockercfg-xksvg\"" Oct 29 23:46:55.593099 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.592915 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6"] Oct 29 23:46:55.632230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.632177 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-token\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.632478 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.632247 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-envoy\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.632478 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.632291 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istiod-ca-cert\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.632478 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.632437 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-workload-certs\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.632744 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.632580 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-podinfo\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.632744 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.632636 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-credential-socket\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.632899 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.632784 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-workload-socket\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.632899 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.632868 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-data\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.633057 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.632916 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x69nx\" (UniqueName: \"kubernetes.io/projected/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-kube-api-access-x69nx\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.734542 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.734499 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x69nx\" (UniqueName: \"kubernetes.io/projected/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-kube-api-access-x69nx\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.734729 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.734681 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-token\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.734729 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.734708 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-envoy\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.734816 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.734733 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istiod-ca-cert\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.734816 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.734795 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-workload-certs\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.734898 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.734852 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-podinfo\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.734950 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.734898 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-credential-socket\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.735019 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.735006 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-workload-socket\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.735101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.735084 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-data\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.735282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.735255 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-workload-certs\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.735388 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.735281 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-credential-socket\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.735459 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.735444 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-workload-socket\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.735635 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.735579 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-data\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.735879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.735854 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istiod-ca-cert\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.737280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.737256 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-envoy\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.737858 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.737835 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-podinfo\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.758052 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.758019 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-token\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.758372 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.758346 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x69nx\" (UniqueName: \"kubernetes.io/projected/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-kube-api-access-x69nx\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:55.873906 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:55.873874 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:56.210425 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:46:56.210392 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de79fc1_b30d_40eb_9f6d_55b1ddde3139.slice/crio-0de7e1e05641f9b1e0294bd7ff9d7f5c46ecc55b1b91f51ae3eb80a21406c188 WatchSource:0}: Error finding container 0de7e1e05641f9b1e0294bd7ff9d7f5c46ecc55b1b91f51ae3eb80a21406c188: Status 404 returned error can't find the container with id 0de7e1e05641f9b1e0294bd7ff9d7f5c46ecc55b1b91f51ae3eb80a21406c188 Oct 29 23:46:56.211811 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:56.211781 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6"] Oct 29 23:46:56.213872 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:56.213843 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:46:56.213944 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:56.213905 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:46:56.213944 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:56.213935 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:46:57.204350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:57.204306 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" event={"ID":"1de79fc1-b30d-40eb-9f6d-55b1ddde3139","Type":"ContainerStarted","Data":"2a3bbad587a0352c4e8e0da33e0b5a41ed4832fb9067b754026e424ef2eb6639"} Oct 29 23:46:57.204885 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:57.204371 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" event={"ID":"1de79fc1-b30d-40eb-9f6d-55b1ddde3139","Type":"ContainerStarted","Data":"0de7e1e05641f9b1e0294bd7ff9d7f5c46ecc55b1b91f51ae3eb80a21406c188"} Oct 29 23:46:57.278717 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:57.278666 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" podStartSLOduration=2.278652774 podStartE2EDuration="2.278652774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:46:57.265548988 +0000 UTC m=+3863.917047520" watchObservedRunningTime="2026-10-29 23:46:57.278652774 +0000 UTC m=+3863.930151305" Oct 29 23:46:57.874926 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:57.874890 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:46:57.876834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:57.876807 5057 patch_prober.go:29] interesting pod/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body= Oct 29 23:46:57.876993 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:57.876854 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" podUID="1de79fc1-b30d-40eb-9f6d-55b1ddde3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" Oct 29 23:46:58.874944 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:58.874800 5057 patch_prober.go:29] interesting pod/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body= Oct 29 23:46:58.874944 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:58.874857 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" podUID="1de79fc1-b30d-40eb-9f6d-55b1ddde3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" Oct 29 23:46:59.874961 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:59.874921 5057 patch_prober.go:29] interesting pod/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body= Oct 29 23:46:59.875492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:46:59.874997 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" podUID="1de79fc1-b30d-40eb-9f6d-55b1ddde3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" Oct 29 23:47:00.874642 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:00.874603 5057 patch_prober.go:29] interesting pod/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body= Oct 29 23:47:00.874834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:00.874670 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" podUID="1de79fc1-b30d-40eb-9f6d-55b1ddde3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" Oct 29 23:47:01.878832 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:01.878795 5057 patch_prober.go:29] interesting pod/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body= Oct 29 23:47:01.879464 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:01.879429 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" podUID="1de79fc1-b30d-40eb-9f6d-55b1ddde3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" Oct 29 23:47:02.874816 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:02.874777 5057 patch_prober.go:29] interesting pod/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body= Oct 29 23:47:02.875004 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:02.874856 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" podUID="1de79fc1-b30d-40eb-9f6d-55b1ddde3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" Oct 29 23:47:03.881863 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:03.881831 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:47:03.882329 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:03.882312 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:47:03.883144 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:03.883120 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:47:03.917036 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:47:03.916996 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:47:33.914783 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:47:33.914743 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:47:35.351251 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:35.351215 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:47:35.368577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:35.368550 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:47:45.994720 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:45.994682 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6"] Oct 29 23:47:45.995282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:45.994935 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" podUID="ba55c571-e46b-4ed5-af12-0cba3e7b44d7" containerName="maas-api" containerID="cri-o://c44b5a118d8df86c648a976f3f3ac7c05bf172ac4e83dee47eb55b05e6bfe382" gracePeriod=30 Oct 29 23:47:46.633076 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:46.633045 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" Oct 29 23:47:46.760049 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:46.760007 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-kube-api-access-hqwms\" (UniqueName: \"kubernetes.io/projected/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-kube-api-access-hqwms\") pod \"ba55c571-e46b-4ed5-af12-0cba3e7b44d7\" (UID: \"ba55c571-e46b-4ed5-af12-0cba3e7b44d7\") " Oct 29 23:47:46.810074 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:46.810037 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-openshift-service-ca\") pod \"ba55c571-e46b-4ed5-af12-0cba3e7b44d7\" (UID: \"ba55c571-e46b-4ed5-af12-0cba3e7b44d7\") " Oct 29 23:47:46.810449 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:46.810421 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-maas-api-tls\") pod \"ba55c571-e46b-4ed5-af12-0cba3e7b44d7\" (UID: \"ba55c571-e46b-4ed5-af12-0cba3e7b44d7\") " Oct 29 23:47:46.810781 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:46.810761 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-metrics-certs\") pod \"ba55c571-e46b-4ed5-af12-0cba3e7b44d7\" (UID: \"ba55c571-e46b-4ed5-af12-0cba3e7b44d7\") " Oct 29 23:47:46.821935 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:46.821600 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-openshift-service-ca" pod "ba55c571-e46b-4ed5-af12-0cba3e7b44d7" (UID: "ba55c571-e46b-4ed5-af12-0cba3e7b44d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:47:46.888682 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:46.888645 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-kube-api-access-hqwms" pod "ba55c571-e46b-4ed5-af12-0cba3e7b44d7" (UID: "ba55c571-e46b-4ed5-af12-0cba3e7b44d7"). InnerVolumeSpecName "kube-api-access-hqwms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:47:46.889045 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:46.889013 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-maas-api-tls" pod "ba55c571-e46b-4ed5-af12-0cba3e7b44d7" (UID: "ba55c571-e46b-4ed5-af12-0cba3e7b44d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:47:46.905396 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:46.905359 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-metrics-certs" pod "ba55c571-e46b-4ed5-af12-0cba3e7b44d7" (UID: "ba55c571-e46b-4ed5-af12-0cba3e7b44d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:47:46.915153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:46.915121 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:47:46.915430 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:46.915412 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:47:46.915551 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:46.915540 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:47:46.915652 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:46.915641 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqwms\" (UniqueName: \"kubernetes.io/projected/ba55c571-e46b-4ed5-af12-0cba3e7b44d7-kube-api-access-hqwms\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:47:46.951852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:46.951816 5057 generic.go:356] "Generic (PLEG): container finished" podID="ba55c571-e46b-4ed5-af12-0cba3e7b44d7" containerID="c44b5a118d8df86c648a976f3f3ac7c05bf172ac4e83dee47eb55b05e6bfe382" exitCode=0 Oct 29 23:47:46.952170 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:46.951867 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" event={"ID":"ba55c571-e46b-4ed5-af12-0cba3e7b44d7","Type":"ContainerDied","Data":"c44b5a118d8df86c648a976f3f3ac7c05bf172ac4e83dee47eb55b05e6bfe382"} Oct 29 23:47:46.952289 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:46.951955 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" Oct 29 23:47:46.958157 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:46.958125 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6" event={"ID":"ba55c571-e46b-4ed5-af12-0cba3e7b44d7","Type":"ContainerDied","Data":"2b4ca57e36c4dd7c00dada2fdd65ca0e22bc7ca4bbd765911edd031b60e69509"} Oct 29 23:47:46.958330 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:46.958178 5057 scope.go:122] "RemoveContainer" containerID="c44b5a118d8df86c648a976f3f3ac7c05bf172ac4e83dee47eb55b05e6bfe382" Oct 29 23:47:46.977044 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:46.977024 5057 scope.go:122] "RemoveContainer" containerID="c44b5a118d8df86c648a976f3f3ac7c05bf172ac4e83dee47eb55b05e6bfe382" Oct 29 23:47:46.977643 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:47:46.977614 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44b5a118d8df86c648a976f3f3ac7c05bf172ac4e83dee47eb55b05e6bfe382\": container with ID starting with c44b5a118d8df86c648a976f3f3ac7c05bf172ac4e83dee47eb55b05e6bfe382 not found: ID does not exist" containerID="c44b5a118d8df86c648a976f3f3ac7c05bf172ac4e83dee47eb55b05e6bfe382" Oct 29 23:47:46.977876 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:46.977837 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44b5a118d8df86c648a976f3f3ac7c05bf172ac4e83dee47eb55b05e6bfe382"} err="failed to get container status \"c44b5a118d8df86c648a976f3f3ac7c05bf172ac4e83dee47eb55b05e6bfe382\": rpc error: code = NotFound desc = could not find container \"c44b5a118d8df86c648a976f3f3ac7c05bf172ac4e83dee47eb55b05e6bfe382\": container with ID starting with c44b5a118d8df86c648a976f3f3ac7c05bf172ac4e83dee47eb55b05e6bfe382 not found: ID does not exist" Oct 29 23:47:47.008869 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:47.008838 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6"] Oct 29 23:47:47.021929 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:47.021774 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b7a567-74dfb757bc-5mzv6"] Oct 29 23:47:47.485378 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:47.485343 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba55c571-e46b-4ed5-af12-0cba3e7b44d7" path="/var/lib/kubelet/pods/ba55c571-e46b-4ed5-af12-0cba3e7b44d7/volumes" Oct 29 23:47:51.953023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:51.952989 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr"] Oct 29 23:47:51.966210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:51.953842 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" podUID="895c9cf8-a426-45b5-b9b8-ea94ad9b407b" containerName="istio-proxy" containerID="cri-o://91e19ca045ac4f61702f9f1624d8db945a70a09737883b3f2683be7c3f49f3a8" gracePeriod=30 Oct 29 23:47:54.509685 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:54.509644 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd"] Oct 29 23:47:54.514032 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:54.511284 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" podUID="72f6fe5b-dae2-4624-b4f4-bb329f370b87" containerName="istio-proxy" containerID="cri-o://4d58f54a6ab5734974a78e6a15eb75dda3fb83c69148dec68f7ea241db5a3f44" gracePeriod=30 Oct 29 23:47:55.170109 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.170074 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4"] Oct 29 23:47:55.171051 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.171033 5057 cpu_manager.go:414] "containerMap: removing container" podUID="ba55c571-e46b-4ed5-af12-0cba3e7b44d7" containerName="maas-api" Oct 29 23:47:55.171126 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.171053 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba55c571-e46b-4ed5-af12-0cba3e7b44d7" containerName="maas-api" Oct 29 23:47:55.171222 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.171178 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba55c571-e46b-4ed5-af12-0cba3e7b44d7" containerName="maas-api" Oct 29 23:47:55.177039 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.177016 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.211399 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.211363 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4"] Oct 29 23:47:55.226531 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.226489 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-750972-openshift-default-dockercfg-7wjmt\"" Oct 29 23:47:55.296169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.296131 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-credential-socket\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.296365 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.296296 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh4jc\" (UniqueName: \"kubernetes.io/projected/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-kube-api-access-wh4jc\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.296422 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.296407 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-data\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.296467 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.296441 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-workload-socket\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.296508 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.296466 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istiod-ca-cert\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.296686 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.296665 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-token\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.296788 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.296705 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-envoy\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.296851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.296818 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-podinfo\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.297001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.296985 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-workload-certs\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.398857 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.398817 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-data\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.399057 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.398870 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-workload-socket\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.399057 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.398896 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istiod-ca-cert\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.399057 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.399019 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-token\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.399272 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.399064 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-envoy\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.399272 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.399116 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-podinfo\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.399392 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.399330 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-workload-certs\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.399392 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.399352 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-data\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.399511 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.399396 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-workload-socket\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.399511 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.399416 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-credential-socket\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.399632 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.399523 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh4jc\" (UniqueName: \"kubernetes.io/projected/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-kube-api-access-wh4jc\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.399752 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.399668 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-workload-certs\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.399752 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.399670 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-credential-socket\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.399842 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.399776 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istiod-ca-cert\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.401749 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.401724 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-envoy\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.401861 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.401768 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-podinfo\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.423108 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.423048 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh4jc\" (UniqueName: \"kubernetes.io/projected/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-kube-api-access-wh4jc\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.423108 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.423060 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-token\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.491973 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.491945 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:55.828232 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.828202 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4"] Oct 29 23:47:55.829116 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:47:55.829088 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b47a487_a766_4e9b_b3d4_c9c94e4c8cdc.slice/crio-4fedd2f3a1075ca6205b17cfd15ef1a5bda1fe52722d8aa3a808b61f27ced2b8 WatchSource:0}: Error finding container 4fedd2f3a1075ca6205b17cfd15ef1a5bda1fe52722d8aa3a808b61f27ced2b8: Status 404 returned error can't find the container with id 4fedd2f3a1075ca6205b17cfd15ef1a5bda1fe52722d8aa3a808b61f27ced2b8 Oct 29 23:47:55.832670 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.832630 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:47:55.832793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.832711 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:47:55.832793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:55.832752 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:47:56.416897 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:56.416786 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" event={"ID":"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc","Type":"ContainerStarted","Data":"98fdff5627f6a62754cdbbc651f18a212148fcc5c5f306081792629854cf72e2"} Oct 29 23:47:56.416897 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:56.416857 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" event={"ID":"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc","Type":"ContainerStarted","Data":"4fedd2f3a1075ca6205b17cfd15ef1a5bda1fe52722d8aa3a808b61f27ced2b8"} Oct 29 23:47:56.461735 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:56.461699 5057 patch_prober.go:29] interesting pod/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:47:56.461927 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:56.461749 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" podUID="72f6fe5b-dae2-4624-b4f4-bb329f370b87" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:47:56.478086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:56.478023 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" podStartSLOduration=1.478004077 podStartE2EDuration="1.478004077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:47:56.473768218 +0000 UTC m=+3923.125266748" watchObservedRunningTime="2026-10-29 23:47:56.478004077 +0000 UTC m=+3923.129502612" Oct 29 23:47:56.493037 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:56.493001 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:47:56.494674 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:56.494646 5057 patch_prober.go:29] interesting pod/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Oct 29 23:47:56.494804 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:56.494692 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" podUID="3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Oct 29 23:47:57.485445 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.483549 5057 generic.go:356] "Generic (PLEG): container finished" podID="895c9cf8-a426-45b5-b9b8-ea94ad9b407b" containerID="91e19ca045ac4f61702f9f1624d8db945a70a09737883b3f2683be7c3f49f3a8" exitCode=0 Oct 29 23:47:57.493217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.492407 5057 patch_prober.go:29] interesting pod/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Oct 29 23:47:57.493217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.492463 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" podUID="3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Oct 29 23:47:57.506220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.505628 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" event={"ID":"895c9cf8-a426-45b5-b9b8-ea94ad9b407b","Type":"ContainerDied","Data":"91e19ca045ac4f61702f9f1624d8db945a70a09737883b3f2683be7c3f49f3a8"} Oct 29 23:47:57.506220 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.505777 5057 scope.go:122] "RemoveContainer" containerID="39e546bcbd91f4dd08cfb867bc69d11f7836bb85153515934df0e8933f3897e3" Oct 29 23:47:57.670854 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.670809 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:47:57.787801 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.787765 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istiod-ca-cert\") pod \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " Oct 29 23:47:57.787974 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.787815 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-data\") pod \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " Oct 29 23:47:57.787974 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.787858 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-credential-socket\") pod \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " Oct 29 23:47:57.787974 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.787914 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-workload-socket\") pod \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " Oct 29 23:47:57.787974 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.787947 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-token\" (UniqueName: \"kubernetes.io/projected/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-token\") pod \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " Oct 29 23:47:57.788119 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.788058 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-workload-certs\") pod \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " Oct 29 23:47:57.788154 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.788141 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-kube-api-access-jdw9j\" (UniqueName: \"kubernetes.io/projected/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-kube-api-access-jdw9j\") pod \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " Oct 29 23:47:57.788266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.788249 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-podinfo\") pod \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " Oct 29 23:47:57.788435 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.788420 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-envoy\") pod \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\" (UID: \"895c9cf8-a426-45b5-b9b8-ea94ad9b407b\") " Oct 29 23:47:57.789096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.788742 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-workload-certs" pod "895c9cf8-a426-45b5-b9b8-ea94ad9b407b" (UID: "895c9cf8-a426-45b5-b9b8-ea94ad9b407b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:47:57.789217 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.789144 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-workload-socket" pod "895c9cf8-a426-45b5-b9b8-ea94ad9b407b" (UID: "895c9cf8-a426-45b5-b9b8-ea94ad9b407b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:47:57.789695 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.789630 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:47:57.789695 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.789658 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:47:57.789924 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.789898 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-data" pod "895c9cf8-a426-45b5-b9b8-ea94ad9b407b" (UID: "895c9cf8-a426-45b5-b9b8-ea94ad9b407b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:47:57.790502 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.790484 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istiod-ca-cert" pod "895c9cf8-a426-45b5-b9b8-ea94ad9b407b" (UID: "895c9cf8-a426-45b5-b9b8-ea94ad9b407b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:47:57.790657 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.790642 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-credential-socket" pod "895c9cf8-a426-45b5-b9b8-ea94ad9b407b" (UID: "895c9cf8-a426-45b5-b9b8-ea94ad9b407b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:47:57.791756 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.791652 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-token" pod "895c9cf8-a426-45b5-b9b8-ea94ad9b407b" (UID: "895c9cf8-a426-45b5-b9b8-ea94ad9b407b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:47:57.794396 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.794370 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-kube-api-access-jdw9j" pod "895c9cf8-a426-45b5-b9b8-ea94ad9b407b" (UID: "895c9cf8-a426-45b5-b9b8-ea94ad9b407b"). InnerVolumeSpecName "kube-api-access-jdw9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:47:57.794711 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.794684 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-envoy" pod "895c9cf8-a426-45b5-b9b8-ea94ad9b407b" (UID: "895c9cf8-a426-45b5-b9b8-ea94ad9b407b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:47:57.796512 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.796493 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-podinfo" pod "895c9cf8-a426-45b5-b9b8-ea94ad9b407b" (UID: "895c9cf8-a426-45b5-b9b8-ea94ad9b407b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:47:57.917397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.917308 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:47:57.917635 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.917615 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:47:57.917752 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.917739 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:47:57.917827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.917818 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:47:57.917880 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.917872 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdw9j\" (UniqueName: \"kubernetes.io/projected/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-kube-api-access-jdw9j\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:47:57.917934 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.917927 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:47:57.917987 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:57.917981 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895c9cf8-a426-45b5-b9b8-ea94ad9b407b-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:47:58.492440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:58.492401 5057 patch_prober.go:29] interesting pod/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Oct 29 23:47:58.492896 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:58.492459 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" podUID="3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Oct 29 23:47:58.493748 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:58.493722 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" Oct 29 23:47:58.493879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:58.493761 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr" event={"ID":"895c9cf8-a426-45b5-b9b8-ea94ad9b407b","Type":"ContainerDied","Data":"7630e51a0407a4de29d5b740a267ea9c9619ca88c6e4c997b2442c1a74c73800"} Oct 29 23:47:58.493879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:58.493818 5057 scope.go:122] "RemoveContainer" containerID="91e19ca045ac4f61702f9f1624d8db945a70a09737883b3f2683be7c3f49f3a8" Oct 29 23:47:58.574569 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:58.574322 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr"] Oct 29 23:47:58.579433 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:58.579403 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-k4cjr"] Oct 29 23:47:59.484531 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.484497 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="895c9cf8-a426-45b5-b9b8-ea94ad9b407b" path="/var/lib/kubelet/pods/895c9cf8-a426-45b5-b9b8-ea94ad9b407b/volumes" Oct 29 23:47:59.492665 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.492642 5057 patch_prober.go:29] interesting pod/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Oct 29 23:47:59.493007 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.492686 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" podUID="3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Oct 29 23:47:59.902975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.902948 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:47:59.981140 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.981107 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-podinfo\") pod \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " Oct 29 23:47:59.981362 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.981173 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-token\" (UniqueName: \"kubernetes.io/projected/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-token\") pod \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " Oct 29 23:47:59.981362 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.981229 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-workload-certs\") pod \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " Oct 29 23:47:59.981362 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.981345 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f6fe5b-dae2-4624-b4f4-bb329f370b87-kube-api-access-7trbp\" (UniqueName: \"kubernetes.io/projected/72f6fe5b-dae2-4624-b4f4-bb329f370b87-kube-api-access-7trbp\") pod \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " Oct 29 23:47:59.981566 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.981443 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-credential-socket\") pod \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " Oct 29 23:47:59.981566 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.981470 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-workload-socket\") pod \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " Oct 29 23:47:59.981566 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.981498 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-envoy\") pod \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " Oct 29 23:47:59.981566 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.981565 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-data\") pod \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " Oct 29 23:47:59.981773 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.981599 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istiod-ca-cert\") pod \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\" (UID: \"72f6fe5b-dae2-4624-b4f4-bb329f370b87\") " Oct 29 23:47:59.983500 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.983467 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-workload-certs" pod "72f6fe5b-dae2-4624-b4f4-bb329f370b87" (UID: "72f6fe5b-dae2-4624-b4f4-bb329f370b87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:47:59.983710 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.983688 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-data" pod "72f6fe5b-dae2-4624-b4f4-bb329f370b87" (UID: "72f6fe5b-dae2-4624-b4f4-bb329f370b87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:47:59.984136 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.983993 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-workload-socket" pod "72f6fe5b-dae2-4624-b4f4-bb329f370b87" (UID: "72f6fe5b-dae2-4624-b4f4-bb329f370b87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:47:59.984720 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.984684 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-credential-socket" pod "72f6fe5b-dae2-4624-b4f4-bb329f370b87" (UID: "72f6fe5b-dae2-4624-b4f4-bb329f370b87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:47:59.984941 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.984915 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istiod-ca-cert" pod "72f6fe5b-dae2-4624-b4f4-bb329f370b87" (UID: "72f6fe5b-dae2-4624-b4f4-bb329f370b87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:47:59.986936 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.986908 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-token" pod "72f6fe5b-dae2-4624-b4f4-bb329f370b87" (UID: "72f6fe5b-dae2-4624-b4f4-bb329f370b87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:47:59.993221 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.990624 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f6fe5b-dae2-4624-b4f4-bb329f370b87-kube-api-access-7trbp" pod "72f6fe5b-dae2-4624-b4f4-bb329f370b87" (UID: "72f6fe5b-dae2-4624-b4f4-bb329f370b87"). InnerVolumeSpecName "kube-api-access-7trbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:47:59.993620 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.993602 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-envoy" pod "72f6fe5b-dae2-4624-b4f4-bb329f370b87" (UID: "72f6fe5b-dae2-4624-b4f4-bb329f370b87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:47:59.996976 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:47:59.996953 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-podinfo" pod "72f6fe5b-dae2-4624-b4f4-bb329f370b87" (UID: "72f6fe5b-dae2-4624-b4f4-bb329f370b87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:48:00.093585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:00.090965 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:00.093585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:00.091001 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:00.093585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:00.091015 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:00.093585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:00.091028 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7trbp\" (UniqueName: \"kubernetes.io/projected/72f6fe5b-dae2-4624-b4f4-bb329f370b87-kube-api-access-7trbp\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:00.093585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:00.091042 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:00.093585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:00.091056 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:00.093585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:00.091070 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:00.093585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:00.091083 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:00.093585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:00.091096 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f6fe5b-dae2-4624-b4f4-bb329f370b87-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:00.493034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:00.492994 5057 patch_prober.go:29] interesting pod/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Oct 29 23:48:00.493566 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:00.493064 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" podUID="3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Oct 29 23:48:00.512271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:00.512240 5057 generic.go:356] "Generic (PLEG): container finished" podID="72f6fe5b-dae2-4624-b4f4-bb329f370b87" containerID="4d58f54a6ab5734974a78e6a15eb75dda3fb83c69148dec68f7ea241db5a3f44" exitCode=0 Oct 29 23:48:00.512449 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:00.512290 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" Oct 29 23:48:00.512449 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:00.512399 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" event={"ID":"72f6fe5b-dae2-4624-b4f4-bb329f370b87","Type":"ContainerDied","Data":"4d58f54a6ab5734974a78e6a15eb75dda3fb83c69148dec68f7ea241db5a3f44"} Oct 29 23:48:00.512449 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:00.512435 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd" event={"ID":"72f6fe5b-dae2-4624-b4f4-bb329f370b87","Type":"ContainerDied","Data":"c128c5cb0a95045ad4fad085de230a9e49e3ae27ed087e1c1699ba70e70badea"} Oct 29 23:48:00.512578 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:00.512465 5057 scope.go:122] "RemoveContainer" containerID="4d58f54a6ab5734974a78e6a15eb75dda3fb83c69148dec68f7ea241db5a3f44" Oct 29 23:48:00.553743 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:00.553708 5057 scope.go:122] "RemoveContainer" containerID="4d58f54a6ab5734974a78e6a15eb75dda3fb83c69148dec68f7ea241db5a3f44" Oct 29 23:48:00.554292 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:48:00.554269 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d58f54a6ab5734974a78e6a15eb75dda3fb83c69148dec68f7ea241db5a3f44\": container with ID starting with 4d58f54a6ab5734974a78e6a15eb75dda3fb83c69148dec68f7ea241db5a3f44 not found: ID does not exist" containerID="4d58f54a6ab5734974a78e6a15eb75dda3fb83c69148dec68f7ea241db5a3f44" Oct 29 23:48:00.554386 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:00.554302 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d58f54a6ab5734974a78e6a15eb75dda3fb83c69148dec68f7ea241db5a3f44"} err="failed to get container status \"4d58f54a6ab5734974a78e6a15eb75dda3fb83c69148dec68f7ea241db5a3f44\": rpc error: code = NotFound desc = could not find container \"4d58f54a6ab5734974a78e6a15eb75dda3fb83c69148dec68f7ea241db5a3f44\": container with ID starting with 4d58f54a6ab5734974a78e6a15eb75dda3fb83c69148dec68f7ea241db5a3f44 not found: ID does not exist" Oct 29 23:48:00.580401 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:00.580369 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd"] Oct 29 23:48:00.591976 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:00.591948 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-mmxxd"] Oct 29 23:48:01.490689 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:01.490656 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f6fe5b-dae2-4624-b4f4-bb329f370b87" path="/var/lib/kubelet/pods/72f6fe5b-dae2-4624-b4f4-bb329f370b87/volumes" Oct 29 23:48:01.492500 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:01.492470 5057 patch_prober.go:29] interesting pod/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Oct 29 23:48:01.492655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:01.492525 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" podUID="3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Oct 29 23:48:02.498251 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:02.498177 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:48:02.498674 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:02.498636 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:48:02.499597 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:02.499575 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:48:03.915691 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:48:03.915653 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:48:05.470353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:05.470314 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf"] Oct 29 23:48:05.506726 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:05.506693 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs"] Oct 29 23:48:10.471340 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:10.471285 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf" podUID="d0787860-0815-4f18-9424-cd3ab9944810" containerName="payload-processing" containerID="cri-o://dec44c7059e947903408bdb9c4742e11b9028d53e21517fc0d6d4200ac218879" gracePeriod=25 Oct 29 23:48:10.507772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:10.507734 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs" podUID="1eaa234f-64b4-4698-bf92-c0c03de0e701" containerName="payload-pre-processing" containerID="cri-o://75e0fcab00d9a748a17ec1fd29df30a2973fba70513d57ac4f91a2c3daa502e3" gracePeriod=25 Oct 29 23:48:13.824939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:13.824883 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2"] Oct 29 23:48:13.825899 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:13.825867 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" podUID="fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" containerName="istio-proxy" containerID="cri-o://ec9191720004d77e324ed60f793e094da40056f13c29439c5012d9acd9e1ece2" gracePeriod=30 Oct 29 23:48:14.417794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:14.417754 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf" podUID="d0787860-0815-4f18-9424-cd3ab9944810" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:48:14.454210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:14.435524 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs" podUID="1eaa234f-64b4-4698-bf92-c0c03de0e701" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:48:15.463817 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:15.463783 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc"] Oct 29 23:48:15.471415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:15.471372 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" podUID="5d1794a4-da94-40aa-80a4-9d62d8a809ca" containerName="maas-api" containerID="cri-o://813038b0fd11954f359d9a2070b1363c75176b239b3fbc5dfb2cdedbfa835709" gracePeriod=30 Oct 29 23:48:15.946816 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:15.946776 5057 generic.go:356] "Generic (PLEG): container finished" podID="5d1794a4-da94-40aa-80a4-9d62d8a809ca" containerID="813038b0fd11954f359d9a2070b1363c75176b239b3fbc5dfb2cdedbfa835709" exitCode=0 Oct 29 23:48:15.947019 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:15.946850 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" event={"ID":"5d1794a4-da94-40aa-80a4-9d62d8a809ca","Type":"ContainerDied","Data":"813038b0fd11954f359d9a2070b1363c75176b239b3fbc5dfb2cdedbfa835709"} Oct 29 23:48:16.500254 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:16.500161 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" Oct 29 23:48:16.619564 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:16.619532 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d1794a4-da94-40aa-80a4-9d62d8a809ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d1794a4-da94-40aa-80a4-9d62d8a809ca-metrics-certs\") pod \"5d1794a4-da94-40aa-80a4-9d62d8a809ca\" (UID: \"5d1794a4-da94-40aa-80a4-9d62d8a809ca\") " Oct 29 23:48:16.619730 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:16.619648 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d1794a4-da94-40aa-80a4-9d62d8a809ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1794a4-da94-40aa-80a4-9d62d8a809ca-openshift-service-ca\") pod \"5d1794a4-da94-40aa-80a4-9d62d8a809ca\" (UID: \"5d1794a4-da94-40aa-80a4-9d62d8a809ca\") " Oct 29 23:48:16.619730 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:16.619727 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1794a4-da94-40aa-80a4-9d62d8a809ca-kube-api-access-p8rmr\" (UniqueName: \"kubernetes.io/projected/5d1794a4-da94-40aa-80a4-9d62d8a809ca-kube-api-access-p8rmr\") pod \"5d1794a4-da94-40aa-80a4-9d62d8a809ca\" (UID: \"5d1794a4-da94-40aa-80a4-9d62d8a809ca\") " Oct 29 23:48:16.619935 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:16.619920 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d1794a4-da94-40aa-80a4-9d62d8a809ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d1794a4-da94-40aa-80a4-9d62d8a809ca-maas-api-tls\") pod \"5d1794a4-da94-40aa-80a4-9d62d8a809ca\" (UID: \"5d1794a4-da94-40aa-80a4-9d62d8a809ca\") " Oct 29 23:48:16.620034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:16.620007 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1794a4-da94-40aa-80a4-9d62d8a809ca-openshift-service-ca" pod "5d1794a4-da94-40aa-80a4-9d62d8a809ca" (UID: "5d1794a4-da94-40aa-80a4-9d62d8a809ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:48:16.620637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:16.620612 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1794a4-da94-40aa-80a4-9d62d8a809ca-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:16.622160 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:16.622137 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1794a4-da94-40aa-80a4-9d62d8a809ca-kube-api-access-p8rmr" pod "5d1794a4-da94-40aa-80a4-9d62d8a809ca" (UID: "5d1794a4-da94-40aa-80a4-9d62d8a809ca"). InnerVolumeSpecName "kube-api-access-p8rmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:16.622309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:16.622266 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d1794a4-da94-40aa-80a4-9d62d8a809ca-maas-api-tls" pod "5d1794a4-da94-40aa-80a4-9d62d8a809ca" (UID: "5d1794a4-da94-40aa-80a4-9d62d8a809ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:48:16.626590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:16.626561 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d1794a4-da94-40aa-80a4-9d62d8a809ca-metrics-certs" pod "5d1794a4-da94-40aa-80a4-9d62d8a809ca" (UID: "5d1794a4-da94-40aa-80a4-9d62d8a809ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:48:16.722485 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:16.722441 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d1794a4-da94-40aa-80a4-9d62d8a809ca-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:16.722485 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:16.722482 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d1794a4-da94-40aa-80a4-9d62d8a809ca-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:16.722710 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:16.722496 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8rmr\" (UniqueName: \"kubernetes.io/projected/5d1794a4-da94-40aa-80a4-9d62d8a809ca-kube-api-access-p8rmr\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:16.957480 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:16.957448 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" Oct 29 23:48:16.957480 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:16.957459 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc" event={"ID":"5d1794a4-da94-40aa-80a4-9d62d8a809ca","Type":"ContainerDied","Data":"1a458933369bbd2e94d417fdf2b9b9ffadfc87e5bb35b5c73b7227a8aa16c4c2"} Oct 29 23:48:16.957747 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:16.957519 5057 scope.go:122] "RemoveContainer" containerID="813038b0fd11954f359d9a2070b1363c75176b239b3fbc5dfb2cdedbfa835709" Oct 29 23:48:17.014291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:17.014261 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc"] Oct 29 23:48:17.090883 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:17.090831 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-26a2a4-7d8c79d9fb-gg5qc"] Oct 29 23:48:17.483867 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:17.483835 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d1794a4-da94-40aa-80a4-9d62d8a809ca" path="/var/lib/kubelet/pods/5d1794a4-da94-40aa-80a4-9d62d8a809ca/volumes" Oct 29 23:48:18.996384 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:18.996349 5057 generic.go:356] "Generic (PLEG): container finished" podID="fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" containerID="ec9191720004d77e324ed60f793e094da40056f13c29439c5012d9acd9e1ece2" exitCode=0 Oct 29 23:48:18.996975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:18.996536 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" event={"ID":"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31","Type":"ContainerDied","Data":"ec9191720004d77e324ed60f793e094da40056f13c29439c5012d9acd9e1ece2"} Oct 29 23:48:18.997151 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:18.997139 5057 scope.go:122] "RemoveContainer" containerID="8329e3fc2f1ef0936c81518e85af99bc6108bb21fca5b0a93f0457bca99f0dda" Oct 29 23:48:19.201522 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.201493 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:48:19.277026 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.276989 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-data\") pod \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " Oct 29 23:48:19.277241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.277216 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-kube-api-access-hs7pf\" (UniqueName: \"kubernetes.io/projected/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-kube-api-access-hs7pf\") pod \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " Oct 29 23:48:19.277334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.277264 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-workload-socket\") pod \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " Oct 29 23:48:19.277400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.277364 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istiod-ca-cert\") pod \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " Oct 29 23:48:19.277468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.277434 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-workload-certs\") pod \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " Oct 29 23:48:19.277525 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.277473 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-token\" (UniqueName: \"kubernetes.io/projected/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-token\") pod \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " Oct 29 23:48:19.277747 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.277721 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-podinfo\") pod \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " Oct 29 23:48:19.277853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.277794 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-envoy\") pod \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " Oct 29 23:48:19.277853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.277830 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-credential-socket\") pod \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\" (UID: \"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31\") " Oct 29 23:48:19.278439 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.278413 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-workload-certs" pod "fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" (UID: "fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:19.278802 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.278776 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-credential-socket" pod "fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" (UID: "fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:19.278927 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.278784 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:19.279490 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.279452 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istiod-ca-cert" pod "fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" (UID: "fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:48:19.279490 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.279462 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-data" pod "fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" (UID: "fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:19.281056 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.281028 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-workload-socket" pod "fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" (UID: "fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:19.282293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.282267 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-token" pod "fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" (UID: "fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:19.284493 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.284458 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-envoy" pod "fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" (UID: "fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:19.285384 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.285364 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-kube-api-access-hs7pf" pod "fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" (UID: "fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31"). InnerVolumeSpecName "kube-api-access-hs7pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:19.286080 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.286056 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-podinfo" pod "fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" (UID: "fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:48:19.380919 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.380884 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:19.380919 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.380917 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs7pf\" (UniqueName: \"kubernetes.io/projected/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-kube-api-access-hs7pf\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:19.381131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.380932 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:19.381131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.380946 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:19.381131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.380958 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:19.381131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.380968 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:19.381131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.380978 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:19.381131 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:19.380991 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:20.006731 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:20.006700 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" Oct 29 23:48:20.007172 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:20.006740 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2" event={"ID":"fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31","Type":"ContainerDied","Data":"6d8e351c2cb463c0b0aa963db95eabd89ae05763c63563d9835200af1b1f29d2"} Oct 29 23:48:20.007172 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:20.006807 5057 scope.go:122] "RemoveContainer" containerID="ec9191720004d77e324ed60f793e094da40056f13c29439c5012d9acd9e1ece2" Oct 29 23:48:20.075653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:20.075621 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2"] Oct 29 23:48:20.087100 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:20.087065 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-qsrr2"] Oct 29 23:48:21.484871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:21.484833 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" path="/var/lib/kubelet/pods/fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31/volumes" Oct 29 23:48:22.970470 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:22.970435 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:48:23.595360 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:23.595326 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4"] Oct 29 23:48:23.595640 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:23.595614 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" podUID="fd6e00d3-c546-4fec-82fd-98c420fcca2d" containerName="istio-proxy" containerID="cri-o://74d3c6f4e38639e4b3e9dc8b3b013a5bcb4ff5d233bfe8f351a3cf62ca81a823" gracePeriod=30 Oct 29 23:48:24.471211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:24.450353 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf" podUID="d0787860-0815-4f18-9424-cd3ab9944810" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:48:24.471211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:24.468347 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs" podUID="1eaa234f-64b4-4698-bf92-c0c03de0e701" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:48:26.140899 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:26.140866 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9"] Oct 29 23:48:26.141573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:26.141108 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" podUID="9c0197f2-eb04-41dc-8a00-a3f97ee557d3" containerName="istio-proxy" containerID="cri-o://684c126d3c918ca8b158bd1e47d411d60f577473784a2e82fee629d9e9669113" gracePeriod=30 Oct 29 23:48:27.089817 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:27.089761 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7"] Oct 29 23:48:27.090059 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:27.090026 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" podUID="88075736-9838-476b-8cdd-36014547ca59" containerName="maas-api" containerID="cri-o://48b808432e96653ca28663b536501484881cccb44cb4f07b7b053821852a72f4" gracePeriod=30 Oct 29 23:48:27.624456 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:27.624412 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" Oct 29 23:48:27.692369 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:27.692334 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88075736-9838-476b-8cdd-36014547ca59-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88075736-9838-476b-8cdd-36014547ca59-openshift-service-ca\") pod \"88075736-9838-476b-8cdd-36014547ca59\" (UID: \"88075736-9838-476b-8cdd-36014547ca59\") " Oct 29 23:48:27.693535 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:27.693509 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88075736-9838-476b-8cdd-36014547ca59-metrics-certs\" (UniqueName: \"kubernetes.io/secret/88075736-9838-476b-8cdd-36014547ca59-metrics-certs\") pod \"88075736-9838-476b-8cdd-36014547ca59\" (UID: \"88075736-9838-476b-8cdd-36014547ca59\") " Oct 29 23:48:27.709964 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:27.709935 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88075736-9838-476b-8cdd-36014547ca59-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88075736-9838-476b-8cdd-36014547ca59-maas-api-tls\") pod \"88075736-9838-476b-8cdd-36014547ca59\" (UID: \"88075736-9838-476b-8cdd-36014547ca59\") " Oct 29 23:48:27.710663 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:27.710641 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88075736-9838-476b-8cdd-36014547ca59-kube-api-access-slmps\" (UniqueName: \"kubernetes.io/projected/88075736-9838-476b-8cdd-36014547ca59-kube-api-access-slmps\") pod \"88075736-9838-476b-8cdd-36014547ca59\" (UID: \"88075736-9838-476b-8cdd-36014547ca59\") " Oct 29 23:48:27.729262 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:27.693400 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88075736-9838-476b-8cdd-36014547ca59-openshift-service-ca" pod "88075736-9838-476b-8cdd-36014547ca59" (UID: "88075736-9838-476b-8cdd-36014547ca59"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:48:27.774590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:27.774412 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88075736-9838-476b-8cdd-36014547ca59-maas-api-tls" pod "88075736-9838-476b-8cdd-36014547ca59" (UID: "88075736-9838-476b-8cdd-36014547ca59"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:48:27.774590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:27.774453 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88075736-9838-476b-8cdd-36014547ca59-kube-api-access-slmps" pod "88075736-9838-476b-8cdd-36014547ca59" (UID: "88075736-9838-476b-8cdd-36014547ca59"). InnerVolumeSpecName "kube-api-access-slmps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:27.774590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:27.774515 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88075736-9838-476b-8cdd-36014547ca59-metrics-certs" pod "88075736-9838-476b-8cdd-36014547ca59" (UID: "88075736-9838-476b-8cdd-36014547ca59"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:48:27.839522 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:27.831337 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88075736-9838-476b-8cdd-36014547ca59-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:27.839522 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:27.831390 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88075736-9838-476b-8cdd-36014547ca59-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:27.839522 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:27.831412 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88075736-9838-476b-8cdd-36014547ca59-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:27.839522 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:27.831432 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slmps\" (UniqueName: \"kubernetes.io/projected/88075736-9838-476b-8cdd-36014547ca59-kube-api-access-slmps\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:28.253784 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.253726 5057 generic.go:356] "Generic (PLEG): container finished" podID="88075736-9838-476b-8cdd-36014547ca59" containerID="48b808432e96653ca28663b536501484881cccb44cb4f07b7b053821852a72f4" exitCode=0 Oct 29 23:48:28.254045 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.254017 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" Oct 29 23:48:28.254293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.254067 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" event={"ID":"88075736-9838-476b-8cdd-36014547ca59","Type":"ContainerDied","Data":"48b808432e96653ca28663b536501484881cccb44cb4f07b7b053821852a72f4"} Oct 29 23:48:28.254293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.254139 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7" event={"ID":"88075736-9838-476b-8cdd-36014547ca59","Type":"ContainerDied","Data":"5a1cb6a986d9e19677deb9de9b7edc86732e9e89a138c6ab3728b4ef258a0d3c"} Oct 29 23:48:28.254293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.254173 5057 scope.go:122] "RemoveContainer" containerID="48b808432e96653ca28663b536501484881cccb44cb4f07b7b053821852a72f4" Oct 29 23:48:28.267345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.267327 5057 scope.go:122] "RemoveContainer" containerID="48b808432e96653ca28663b536501484881cccb44cb4f07b7b053821852a72f4" Oct 29 23:48:28.267663 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:48:28.267642 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48b808432e96653ca28663b536501484881cccb44cb4f07b7b053821852a72f4\": container with ID starting with 48b808432e96653ca28663b536501484881cccb44cb4f07b7b053821852a72f4 not found: ID does not exist" containerID="48b808432e96653ca28663b536501484881cccb44cb4f07b7b053821852a72f4" Oct 29 23:48:28.267722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.267674 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b808432e96653ca28663b536501484881cccb44cb4f07b7b053821852a72f4"} err="failed to get container status \"48b808432e96653ca28663b536501484881cccb44cb4f07b7b053821852a72f4\": rpc error: code = NotFound desc = could not find container \"48b808432e96653ca28663b536501484881cccb44cb4f07b7b053821852a72f4\": container with ID starting with 48b808432e96653ca28663b536501484881cccb44cb4f07b7b053821852a72f4 not found: ID does not exist" Oct 29 23:48:28.300757 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.300729 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:48:28.300883 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.300777 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" podUID="fd6e00d3-c546-4fec-82fd-98c420fcca2d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:48:28.311992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.311960 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7"] Oct 29 23:48:28.321648 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.321620 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-52f67b-6454c598c7-wwvl7"] Oct 29 23:48:28.912677 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.912648 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:48:28.915613 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.915582 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-workload-socket\") pod \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " Oct 29 23:48:28.915783 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.915624 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istiod-ca-cert\") pod \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " Oct 29 23:48:28.915783 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.915726 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-token\" (UniqueName: \"kubernetes.io/projected/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-token\") pod \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " Oct 29 23:48:28.915946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.915800 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-credential-socket\") pod \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " Oct 29 23:48:28.915946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.915831 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-workload-certs\") pod \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " Oct 29 23:48:28.915946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.915850 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-workload-socket" pod "fd6e00d3-c546-4fec-82fd-98c420fcca2d" (UID: "fd6e00d3-c546-4fec-82fd-98c420fcca2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:28.915946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.915874 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-envoy\") pod \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " Oct 29 23:48:28.915946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.915929 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-podinfo\") pod \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " Oct 29 23:48:28.916695 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.915981 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd6e00d3-c546-4fec-82fd-98c420fcca2d-kube-api-access-qjkjv\" (UniqueName: \"kubernetes.io/projected/fd6e00d3-c546-4fec-82fd-98c420fcca2d-kube-api-access-qjkjv\") pod \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " Oct 29 23:48:28.916695 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.916045 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-data\") pod \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\" (UID: \"fd6e00d3-c546-4fec-82fd-98c420fcca2d\") " Oct 29 23:48:28.916695 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.916079 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istiod-ca-cert" pod "fd6e00d3-c546-4fec-82fd-98c420fcca2d" (UID: "fd6e00d3-c546-4fec-82fd-98c420fcca2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:48:28.916695 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.916119 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-workload-certs" pod "fd6e00d3-c546-4fec-82fd-98c420fcca2d" (UID: "fd6e00d3-c546-4fec-82fd-98c420fcca2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:28.916695 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.916325 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-credential-socket" pod "fd6e00d3-c546-4fec-82fd-98c420fcca2d" (UID: "fd6e00d3-c546-4fec-82fd-98c420fcca2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:28.916695 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.916345 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-data" pod "fd6e00d3-c546-4fec-82fd-98c420fcca2d" (UID: "fd6e00d3-c546-4fec-82fd-98c420fcca2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:28.917105 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.917080 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:28.917176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.917111 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:28.917176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.917127 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:28.917176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.917143 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:28.917176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.917158 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:28.918552 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.918527 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6e00d3-c546-4fec-82fd-98c420fcca2d-kube-api-access-qjkjv" pod "fd6e00d3-c546-4fec-82fd-98c420fcca2d" (UID: "fd6e00d3-c546-4fec-82fd-98c420fcca2d"). InnerVolumeSpecName "kube-api-access-qjkjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:28.918737 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.918716 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-envoy" pod "fd6e00d3-c546-4fec-82fd-98c420fcca2d" (UID: "fd6e00d3-c546-4fec-82fd-98c420fcca2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:28.919143 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.919112 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-podinfo" pod "fd6e00d3-c546-4fec-82fd-98c420fcca2d" (UID: "fd6e00d3-c546-4fec-82fd-98c420fcca2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:48:28.919143 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:28.919133 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-token" pod "fd6e00d3-c546-4fec-82fd-98c420fcca2d" (UID: "fd6e00d3-c546-4fec-82fd-98c420fcca2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:29.019348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:29.019304 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjkjv\" (UniqueName: \"kubernetes.io/projected/fd6e00d3-c546-4fec-82fd-98c420fcca2d-kube-api-access-qjkjv\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:29.019348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:29.019342 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:29.019621 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:29.019357 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:29.019621 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:29.019372 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd6e00d3-c546-4fec-82fd-98c420fcca2d-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:29.266577 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:29.266543 5057 generic.go:356] "Generic (PLEG): container finished" podID="fd6e00d3-c546-4fec-82fd-98c420fcca2d" containerID="74d3c6f4e38639e4b3e9dc8b3b013a5bcb4ff5d233bfe8f351a3cf62ca81a823" exitCode=0 Oct 29 23:48:29.266796 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:29.266574 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" event={"ID":"fd6e00d3-c546-4fec-82fd-98c420fcca2d","Type":"ContainerDied","Data":"74d3c6f4e38639e4b3e9dc8b3b013a5bcb4ff5d233bfe8f351a3cf62ca81a823"} Oct 29 23:48:29.266796 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:29.266619 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" Oct 29 23:48:29.266796 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:29.266643 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4" event={"ID":"fd6e00d3-c546-4fec-82fd-98c420fcca2d","Type":"ContainerDied","Data":"59e3e10d5c75e2bb7b3b69fa6193cbf3165afe97486839eb48c08c0163e7c7e6"} Oct 29 23:48:29.266796 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:29.266671 5057 scope.go:122] "RemoveContainer" containerID="74d3c6f4e38639e4b3e9dc8b3b013a5bcb4ff5d233bfe8f351a3cf62ca81a823" Oct 29 23:48:29.301663 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:29.301631 5057 scope.go:122] "RemoveContainer" containerID="59396594e1193249bbbe8a320ddba6f2d133c40082212650848d4e4589440d1d" Oct 29 23:48:29.315498 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:29.315468 5057 scope.go:122] "RemoveContainer" containerID="74d3c6f4e38639e4b3e9dc8b3b013a5bcb4ff5d233bfe8f351a3cf62ca81a823" Oct 29 23:48:29.315856 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:48:29.315824 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d3c6f4e38639e4b3e9dc8b3b013a5bcb4ff5d233bfe8f351a3cf62ca81a823\": container with ID starting with 74d3c6f4e38639e4b3e9dc8b3b013a5bcb4ff5d233bfe8f351a3cf62ca81a823 not found: ID does not exist" containerID="74d3c6f4e38639e4b3e9dc8b3b013a5bcb4ff5d233bfe8f351a3cf62ca81a823" Oct 29 23:48:29.315958 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:29.315867 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d3c6f4e38639e4b3e9dc8b3b013a5bcb4ff5d233bfe8f351a3cf62ca81a823"} err="failed to get container status \"74d3c6f4e38639e4b3e9dc8b3b013a5bcb4ff5d233bfe8f351a3cf62ca81a823\": rpc error: code = NotFound desc = could not find container \"74d3c6f4e38639e4b3e9dc8b3b013a5bcb4ff5d233bfe8f351a3cf62ca81a823\": container with ID starting with 74d3c6f4e38639e4b3e9dc8b3b013a5bcb4ff5d233bfe8f351a3cf62ca81a823 not found: ID does not exist" Oct 29 23:48:29.315958 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:29.315891 5057 scope.go:122] "RemoveContainer" containerID="59396594e1193249bbbe8a320ddba6f2d133c40082212650848d4e4589440d1d" Oct 29 23:48:29.316154 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:48:29.316133 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59396594e1193249bbbe8a320ddba6f2d133c40082212650848d4e4589440d1d\": container with ID starting with 59396594e1193249bbbe8a320ddba6f2d133c40082212650848d4e4589440d1d not found: ID does not exist" containerID="59396594e1193249bbbe8a320ddba6f2d133c40082212650848d4e4589440d1d" Oct 29 23:48:29.316253 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:29.316159 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59396594e1193249bbbe8a320ddba6f2d133c40082212650848d4e4589440d1d"} err="failed to get container status \"59396594e1193249bbbe8a320ddba6f2d133c40082212650848d4e4589440d1d\": rpc error: code = NotFound desc = could not find container \"59396594e1193249bbbe8a320ddba6f2d133c40082212650848d4e4589440d1d\": container with ID starting with 59396594e1193249bbbe8a320ddba6f2d133c40082212650848d4e4589440d1d not found: ID does not exist" Oct 29 23:48:29.343071 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:29.343035 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4"] Oct 29 23:48:29.356687 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:29.356655 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-dea9cb-openshift-default-687dccdb9-znhz4"] Oct 29 23:48:29.485862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:29.485819 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88075736-9838-476b-8cdd-36014547ca59" path="/var/lib/kubelet/pods/88075736-9838-476b-8cdd-36014547ca59/volumes" Oct 29 23:48:29.486487 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:29.486463 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd6e00d3-c546-4fec-82fd-98c420fcca2d" path="/var/lib/kubelet/pods/fd6e00d3-c546-4fec-82fd-98c420fcca2d/volumes" Oct 29 23:48:30.825472 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:30.825441 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf" Oct 29 23:48:30.850444 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:30.850412 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0787860-0815-4f18-9424-cd3ab9944810-kube-api-access-gx6k9\" (UniqueName: \"kubernetes.io/projected/d0787860-0815-4f18-9424-cd3ab9944810-kube-api-access-gx6k9\") pod \"d0787860-0815-4f18-9424-cd3ab9944810\" (UID: \"d0787860-0815-4f18-9424-cd3ab9944810\") " Oct 29 23:48:30.850611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:30.850495 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0787860-0815-4f18-9424-cd3ab9944810-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0787860-0815-4f18-9424-cd3ab9944810-plugins-config-volume\") pod \"d0787860-0815-4f18-9424-cd3ab9944810\" (UID: \"d0787860-0815-4f18-9424-cd3ab9944810\") " Oct 29 23:48:30.851029 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:30.851004 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0787860-0815-4f18-9424-cd3ab9944810-plugins-config-volume" pod "d0787860-0815-4f18-9424-cd3ab9944810" (UID: "d0787860-0815-4f18-9424-cd3ab9944810"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:48:30.851376 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:30.851357 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0787860-0815-4f18-9424-cd3ab9944810-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:30.852660 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:30.852632 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0787860-0815-4f18-9424-cd3ab9944810-kube-api-access-gx6k9" pod "d0787860-0815-4f18-9424-cd3ab9944810" (UID: "d0787860-0815-4f18-9424-cd3ab9944810"). InnerVolumeSpecName "kube-api-access-gx6k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:30.866329 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:30.866305 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs" Oct 29 23:48:30.952334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:30.952297 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eaa234f-64b4-4698-bf92-c0c03de0e701-kube-api-access-7xjj6\" (UniqueName: \"kubernetes.io/projected/1eaa234f-64b4-4698-bf92-c0c03de0e701-kube-api-access-7xjj6\") pod \"1eaa234f-64b4-4698-bf92-c0c03de0e701\" (UID: \"1eaa234f-64b4-4698-bf92-c0c03de0e701\") " Oct 29 23:48:30.952510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:30.952404 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eaa234f-64b4-4698-bf92-c0c03de0e701-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eaa234f-64b4-4698-bf92-c0c03de0e701-plugins-config-volume\") pod \"1eaa234f-64b4-4698-bf92-c0c03de0e701\" (UID: \"1eaa234f-64b4-4698-bf92-c0c03de0e701\") " Oct 29 23:48:30.952809 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:30.952784 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eaa234f-64b4-4698-bf92-c0c03de0e701-plugins-config-volume" pod "1eaa234f-64b4-4698-bf92-c0c03de0e701" (UID: "1eaa234f-64b4-4698-bf92-c0c03de0e701"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:48:30.953263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:30.953242 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eaa234f-64b4-4698-bf92-c0c03de0e701-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:30.953324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:30.953266 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx6k9\" (UniqueName: \"kubernetes.io/projected/d0787860-0815-4f18-9424-cd3ab9944810-kube-api-access-gx6k9\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:30.954676 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:30.954656 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eaa234f-64b4-4698-bf92-c0c03de0e701-kube-api-access-7xjj6" pod "1eaa234f-64b4-4698-bf92-c0c03de0e701" (UID: "1eaa234f-64b4-4698-bf92-c0c03de0e701"). InnerVolumeSpecName "kube-api-access-7xjj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:31.055177 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.055085 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xjj6\" (UniqueName: \"kubernetes.io/projected/1eaa234f-64b4-4698-bf92-c0c03de0e701-kube-api-access-7xjj6\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:31.297683 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.297650 5057 generic.go:356] "Generic (PLEG): container finished" podID="9c0197f2-eb04-41dc-8a00-a3f97ee557d3" containerID="684c126d3c918ca8b158bd1e47d411d60f577473784a2e82fee629d9e9669113" exitCode=0 Oct 29 23:48:31.297876 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.297733 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" event={"ID":"9c0197f2-eb04-41dc-8a00-a3f97ee557d3","Type":"ContainerDied","Data":"684c126d3c918ca8b158bd1e47d411d60f577473784a2e82fee629d9e9669113"} Oct 29 23:48:31.297876 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.297817 5057 scope.go:122] "RemoveContainer" containerID="01e4a75d65e031938da18a02fab0cc284eea8825272860d642633b41a7516a67" Oct 29 23:48:31.299374 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.299348 5057 generic.go:356] "Generic (PLEG): container finished" podID="d0787860-0815-4f18-9424-cd3ab9944810" containerID="dec44c7059e947903408bdb9c4742e11b9028d53e21517fc0d6d4200ac218879" exitCode=0 Oct 29 23:48:31.299500 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.299401 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf" Oct 29 23:48:31.299500 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.299439 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf" event={"ID":"d0787860-0815-4f18-9424-cd3ab9944810","Type":"ContainerDied","Data":"dec44c7059e947903408bdb9c4742e11b9028d53e21517fc0d6d4200ac218879"} Oct 29 23:48:31.299500 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.299497 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf" event={"ID":"d0787860-0815-4f18-9424-cd3ab9944810","Type":"ContainerDied","Data":"5aa4cd2d7cf4703d900300f0bc0a4fede5fa358bd9a9d6d62a692b6708988332"} Oct 29 23:48:31.301378 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.301344 5057 generic.go:356] "Generic (PLEG): container finished" podID="1eaa234f-64b4-4698-bf92-c0c03de0e701" containerID="75e0fcab00d9a748a17ec1fd29df30a2973fba70513d57ac4f91a2c3daa502e3" exitCode=0 Oct 29 23:48:31.301378 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.301372 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs" event={"ID":"1eaa234f-64b4-4698-bf92-c0c03de0e701","Type":"ContainerDied","Data":"75e0fcab00d9a748a17ec1fd29df30a2973fba70513d57ac4f91a2c3daa502e3"} Oct 29 23:48:31.301532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.301389 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs" Oct 29 23:48:31.301532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.301410 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs" event={"ID":"1eaa234f-64b4-4698-bf92-c0c03de0e701","Type":"ContainerDied","Data":"c0e2bc85c438b771a98bbc6ea06d1150be60735ce421f7352f610a644c6a935a"} Oct 29 23:48:31.319532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.319509 5057 scope.go:122] "RemoveContainer" containerID="dec44c7059e947903408bdb9c4742e11b9028d53e21517fc0d6d4200ac218879" Oct 29 23:48:31.330639 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.330619 5057 scope.go:122] "RemoveContainer" containerID="dec44c7059e947903408bdb9c4742e11b9028d53e21517fc0d6d4200ac218879" Oct 29 23:48:31.330914 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:48:31.330889 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec44c7059e947903408bdb9c4742e11b9028d53e21517fc0d6d4200ac218879\": container with ID starting with dec44c7059e947903408bdb9c4742e11b9028d53e21517fc0d6d4200ac218879 not found: ID does not exist" containerID="dec44c7059e947903408bdb9c4742e11b9028d53e21517fc0d6d4200ac218879" Oct 29 23:48:31.331020 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.330927 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec44c7059e947903408bdb9c4742e11b9028d53e21517fc0d6d4200ac218879"} err="failed to get container status \"dec44c7059e947903408bdb9c4742e11b9028d53e21517fc0d6d4200ac218879\": rpc error: code = NotFound desc = could not find container \"dec44c7059e947903408bdb9c4742e11b9028d53e21517fc0d6d4200ac218879\": container with ID starting with dec44c7059e947903408bdb9c4742e11b9028d53e21517fc0d6d4200ac218879 not found: ID does not exist" Oct 29 23:48:31.331020 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.330951 5057 scope.go:122] "RemoveContainer" containerID="75e0fcab00d9a748a17ec1fd29df30a2973fba70513d57ac4f91a2c3daa502e3" Oct 29 23:48:31.341292 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.341272 5057 scope.go:122] "RemoveContainer" containerID="75e0fcab00d9a748a17ec1fd29df30a2973fba70513d57ac4f91a2c3daa502e3" Oct 29 23:48:31.341539 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:48:31.341516 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75e0fcab00d9a748a17ec1fd29df30a2973fba70513d57ac4f91a2c3daa502e3\": container with ID starting with 75e0fcab00d9a748a17ec1fd29df30a2973fba70513d57ac4f91a2c3daa502e3 not found: ID does not exist" containerID="75e0fcab00d9a748a17ec1fd29df30a2973fba70513d57ac4f91a2c3daa502e3" Oct 29 23:48:31.341605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.341549 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75e0fcab00d9a748a17ec1fd29df30a2973fba70513d57ac4f91a2c3daa502e3"} err="failed to get container status \"75e0fcab00d9a748a17ec1fd29df30a2973fba70513d57ac4f91a2c3daa502e3\": rpc error: code = NotFound desc = could not find container \"75e0fcab00d9a748a17ec1fd29df30a2973fba70513d57ac4f91a2c3daa502e3\": container with ID starting with 75e0fcab00d9a748a17ec1fd29df30a2973fba70513d57ac4f91a2c3daa502e3 not found: ID does not exist" Oct 29 23:48:31.359550 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.359523 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf"] Oct 29 23:48:31.363721 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.363691 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-dea9cb-75d9cbcb5b-qbgmf"] Oct 29 23:48:31.397684 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.397610 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs"] Oct 29 23:48:31.401599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.401576 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-dea9cb-5cbc85d855-6vkfs"] Oct 29 23:48:31.486849 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.486813 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eaa234f-64b4-4698-bf92-c0c03de0e701" path="/var/lib/kubelet/pods/1eaa234f-64b4-4698-bf92-c0c03de0e701/volumes" Oct 29 23:48:31.487328 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.487313 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0787860-0815-4f18-9424-cd3ab9944810" path="/var/lib/kubelet/pods/d0787860-0815-4f18-9424-cd3ab9944810/volumes" Oct 29 23:48:31.861284 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.861251 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:48:31.870008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.869981 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-workload-socket\") pod \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " Oct 29 23:48:31.870169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.870014 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-envoy\") pod \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " Oct 29 23:48:31.870169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.870060 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-workload-certs\") pod \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " Oct 29 23:48:31.870169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.870124 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-data\") pod \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " Oct 29 23:48:31.870169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.870167 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-podinfo\") pod \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " Oct 29 23:48:31.870410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.870302 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istiod-ca-cert\") pod \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " Oct 29 23:48:31.870410 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.870342 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-kube-api-access-s6mvx\" (UniqueName: \"kubernetes.io/projected/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-kube-api-access-s6mvx\") pod \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " Oct 29 23:48:31.870526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.870436 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-credential-socket\") pod \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " Oct 29 23:48:31.870526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.870448 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-workload-certs" pod "9c0197f2-eb04-41dc-8a00-a3f97ee557d3" (UID: "9c0197f2-eb04-41dc-8a00-a3f97ee557d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:31.870526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.870471 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-token\" (UniqueName: \"kubernetes.io/projected/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-token\") pod \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\" (UID: \"9c0197f2-eb04-41dc-8a00-a3f97ee557d3\") " Oct 29 23:48:31.870883 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.870855 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-workload-socket" pod "9c0197f2-eb04-41dc-8a00-a3f97ee557d3" (UID: "9c0197f2-eb04-41dc-8a00-a3f97ee557d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:31.871163 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.871141 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:31.871266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.871168 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:31.871592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.871444 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istiod-ca-cert" pod "9c0197f2-eb04-41dc-8a00-a3f97ee557d3" (UID: "9c0197f2-eb04-41dc-8a00-a3f97ee557d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:48:31.871592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.871557 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-credential-socket" pod "9c0197f2-eb04-41dc-8a00-a3f97ee557d3" (UID: "9c0197f2-eb04-41dc-8a00-a3f97ee557d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:31.872226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.872171 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-data" pod "9c0197f2-eb04-41dc-8a00-a3f97ee557d3" (UID: "9c0197f2-eb04-41dc-8a00-a3f97ee557d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:31.872571 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.872552 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-envoy" pod "9c0197f2-eb04-41dc-8a00-a3f97ee557d3" (UID: "9c0197f2-eb04-41dc-8a00-a3f97ee557d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:31.873174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.873150 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-token" pod "9c0197f2-eb04-41dc-8a00-a3f97ee557d3" (UID: "9c0197f2-eb04-41dc-8a00-a3f97ee557d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:31.873886 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.873866 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-podinfo" pod "9c0197f2-eb04-41dc-8a00-a3f97ee557d3" (UID: "9c0197f2-eb04-41dc-8a00-a3f97ee557d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:48:31.874244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.874226 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-kube-api-access-s6mvx" pod "9c0197f2-eb04-41dc-8a00-a3f97ee557d3" (UID: "9c0197f2-eb04-41dc-8a00-a3f97ee557d3"). InnerVolumeSpecName "kube-api-access-s6mvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:31.972424 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.972340 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:31.972602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.972517 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:31.972602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.972540 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:31.972602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.972552 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:31.972602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.972565 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6mvx\" (UniqueName: \"kubernetes.io/projected/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-kube-api-access-s6mvx\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:31.972602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.972578 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:31.972602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:31.972590 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c0197f2-eb04-41dc-8a00-a3f97ee557d3-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:48:32.313890 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:32.313801 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" Oct 29 23:48:32.314042 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:32.313813 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9" event={"ID":"9c0197f2-eb04-41dc-8a00-a3f97ee557d3","Type":"ContainerDied","Data":"e22a909c4a871dee72d637f3020ce7c597313743d178001cb0e178b2603db3d5"} Oct 29 23:48:32.314042 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:32.313955 5057 scope.go:122] "RemoveContainer" containerID="684c126d3c918ca8b158bd1e47d411d60f577473784a2e82fee629d9e9669113" Oct 29 23:48:32.369079 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:32.369048 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9"] Oct 29 23:48:32.382286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:32.382257 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-26a2a4-openshift-default-64db878c67-vrxr9"] Oct 29 23:48:33.485668 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:33.485635 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c0197f2-eb04-41dc-8a00-a3f97ee557d3" path="/var/lib/kubelet/pods/9c0197f2-eb04-41dc-8a00-a3f97ee557d3/volumes" Oct 29 23:48:33.935572 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:48:33.935538 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:48:34.620000 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:34.619959 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx"] Oct 29 23:48:34.741244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:34.737974 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs"] Oct 29 23:48:39.620973 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:39.620927 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx" podUID="1ec6c14d-ff01-457a-abaf-8eab25418539" containerName="payload-processing" containerID="cri-o://cb43dc6f5b6e89126f731f28e5f7e4c3317114aa9e8528cf397854c54cfce3c4" gracePeriod=25 Oct 29 23:48:39.743209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:39.743107 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs" podUID="139df2e5-deb8-44b0-8ed5-fb76650beb06" containerName="payload-pre-processing" containerID="cri-o://820fbfdbeed78f266900f1b1796951e73c390513a8e1338d5618eefca1078777" gracePeriod=25 Oct 29 23:48:42.001618 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:42.001577 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx" podUID="1ec6c14d-ff01-457a-abaf-8eab25418539" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:48:42.002130 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:42.001596 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs" podUID="139df2e5-deb8-44b0-8ed5-fb76650beb06" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:48:51.421087 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:51.421052 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7"] Oct 29 23:48:52.001327 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:52.001287 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs" podUID="139df2e5-deb8-44b0-8ed5-fb76650beb06" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:48:52.001527 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:48:52.001287 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx" podUID="1ec6c14d-ff01-457a-abaf-8eab25418539" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:49:00.059704 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.059667 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs" Oct 29 23:49:00.063677 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.063646 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx" Oct 29 23:49:00.125027 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.124989 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec6c14d-ff01-457a-abaf-8eab25418539-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec6c14d-ff01-457a-abaf-8eab25418539-plugins-config-volume\") pod \"1ec6c14d-ff01-457a-abaf-8eab25418539\" (UID: \"1ec6c14d-ff01-457a-abaf-8eab25418539\") " Oct 29 23:49:00.125308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.125196 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/139df2e5-deb8-44b0-8ed5-fb76650beb06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/139df2e5-deb8-44b0-8ed5-fb76650beb06-plugins-config-volume\") pod \"139df2e5-deb8-44b0-8ed5-fb76650beb06\" (UID: \"139df2e5-deb8-44b0-8ed5-fb76650beb06\") " Oct 29 23:49:00.125308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.125287 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/139df2e5-deb8-44b0-8ed5-fb76650beb06-kube-api-access-vdkzj\" (UniqueName: \"kubernetes.io/projected/139df2e5-deb8-44b0-8ed5-fb76650beb06-kube-api-access-vdkzj\") pod \"139df2e5-deb8-44b0-8ed5-fb76650beb06\" (UID: \"139df2e5-deb8-44b0-8ed5-fb76650beb06\") " Oct 29 23:49:00.125495 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.125331 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec6c14d-ff01-457a-abaf-8eab25418539-kube-api-access-9cbcx\" (UniqueName: \"kubernetes.io/projected/1ec6c14d-ff01-457a-abaf-8eab25418539-kube-api-access-9cbcx\") pod \"1ec6c14d-ff01-457a-abaf-8eab25418539\" (UID: \"1ec6c14d-ff01-457a-abaf-8eab25418539\") " Oct 29 23:49:00.125668 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.125609 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec6c14d-ff01-457a-abaf-8eab25418539-plugins-config-volume" pod "1ec6c14d-ff01-457a-abaf-8eab25418539" (UID: "1ec6c14d-ff01-457a-abaf-8eab25418539"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:49:00.125668 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.125639 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/139df2e5-deb8-44b0-8ed5-fb76650beb06-plugins-config-volume" pod "139df2e5-deb8-44b0-8ed5-fb76650beb06" (UID: "139df2e5-deb8-44b0-8ed5-fb76650beb06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:49:00.126282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.126256 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/139df2e5-deb8-44b0-8ed5-fb76650beb06-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:00.126437 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.126286 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec6c14d-ff01-457a-abaf-8eab25418539-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:00.128273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.128245 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/139df2e5-deb8-44b0-8ed5-fb76650beb06-kube-api-access-vdkzj" pod "139df2e5-deb8-44b0-8ed5-fb76650beb06" (UID: "139df2e5-deb8-44b0-8ed5-fb76650beb06"). InnerVolumeSpecName "kube-api-access-vdkzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:49:00.128452 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.128431 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec6c14d-ff01-457a-abaf-8eab25418539-kube-api-access-9cbcx" pod "1ec6c14d-ff01-457a-abaf-8eab25418539" (UID: "1ec6c14d-ff01-457a-abaf-8eab25418539"). InnerVolumeSpecName "kube-api-access-9cbcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:49:00.227615 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.227578 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdkzj\" (UniqueName: \"kubernetes.io/projected/139df2e5-deb8-44b0-8ed5-fb76650beb06-kube-api-access-vdkzj\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:00.227615 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.227610 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cbcx\" (UniqueName: \"kubernetes.io/projected/1ec6c14d-ff01-457a-abaf-8eab25418539-kube-api-access-9cbcx\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:00.823358 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.823307 5057 generic.go:356] "Generic (PLEG): container finished" podID="1ec6c14d-ff01-457a-abaf-8eab25418539" containerID="cb43dc6f5b6e89126f731f28e5f7e4c3317114aa9e8528cf397854c54cfce3c4" exitCode=0 Oct 29 23:49:00.823548 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.823375 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx" Oct 29 23:49:00.823548 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.823389 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx" event={"ID":"1ec6c14d-ff01-457a-abaf-8eab25418539","Type":"ContainerDied","Data":"cb43dc6f5b6e89126f731f28e5f7e4c3317114aa9e8528cf397854c54cfce3c4"} Oct 29 23:49:00.823548 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.823471 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx" event={"ID":"1ec6c14d-ff01-457a-abaf-8eab25418539","Type":"ContainerDied","Data":"22e9b7995424d334ab5b62b4dfa9c52a7f68841c3cef5d64f19ecd543a20bc54"} Oct 29 23:49:00.823548 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.823503 5057 scope.go:122] "RemoveContainer" containerID="cb43dc6f5b6e89126f731f28e5f7e4c3317114aa9e8528cf397854c54cfce3c4" Oct 29 23:49:00.829448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.829419 5057 generic.go:356] "Generic (PLEG): container finished" podID="139df2e5-deb8-44b0-8ed5-fb76650beb06" containerID="820fbfdbeed78f266900f1b1796951e73c390513a8e1338d5618eefca1078777" exitCode=0 Oct 29 23:49:00.829599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.829476 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs" event={"ID":"139df2e5-deb8-44b0-8ed5-fb76650beb06","Type":"ContainerDied","Data":"820fbfdbeed78f266900f1b1796951e73c390513a8e1338d5618eefca1078777"} Oct 29 23:49:00.829599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.829580 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs" event={"ID":"139df2e5-deb8-44b0-8ed5-fb76650beb06","Type":"ContainerDied","Data":"552799aa90d31e505867a8aa169cca4a5cc9d694e8b410b67d1b343f9670abe2"} Oct 29 23:49:00.829716 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.829487 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs" Oct 29 23:49:00.838479 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.838382 5057 scope.go:122] "RemoveContainer" containerID="cb43dc6f5b6e89126f731f28e5f7e4c3317114aa9e8528cf397854c54cfce3c4" Oct 29 23:49:00.838784 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:49:00.838757 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb43dc6f5b6e89126f731f28e5f7e4c3317114aa9e8528cf397854c54cfce3c4\": container with ID starting with cb43dc6f5b6e89126f731f28e5f7e4c3317114aa9e8528cf397854c54cfce3c4 not found: ID does not exist" containerID="cb43dc6f5b6e89126f731f28e5f7e4c3317114aa9e8528cf397854c54cfce3c4" Oct 29 23:49:00.838871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.838795 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb43dc6f5b6e89126f731f28e5f7e4c3317114aa9e8528cf397854c54cfce3c4"} err="failed to get container status \"cb43dc6f5b6e89126f731f28e5f7e4c3317114aa9e8528cf397854c54cfce3c4\": rpc error: code = NotFound desc = could not find container \"cb43dc6f5b6e89126f731f28e5f7e4c3317114aa9e8528cf397854c54cfce3c4\": container with ID starting with cb43dc6f5b6e89126f731f28e5f7e4c3317114aa9e8528cf397854c54cfce3c4 not found: ID does not exist" Oct 29 23:49:00.838871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.838819 5057 scope.go:122] "RemoveContainer" containerID="820fbfdbeed78f266900f1b1796951e73c390513a8e1338d5618eefca1078777" Oct 29 23:49:00.853959 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.853933 5057 scope.go:122] "RemoveContainer" containerID="820fbfdbeed78f266900f1b1796951e73c390513a8e1338d5618eefca1078777" Oct 29 23:49:00.854566 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:49:00.854532 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"820fbfdbeed78f266900f1b1796951e73c390513a8e1338d5618eefca1078777\": container with ID starting with 820fbfdbeed78f266900f1b1796951e73c390513a8e1338d5618eefca1078777 not found: ID does not exist" containerID="820fbfdbeed78f266900f1b1796951e73c390513a8e1338d5618eefca1078777" Oct 29 23:49:00.854688 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.854576 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"820fbfdbeed78f266900f1b1796951e73c390513a8e1338d5618eefca1078777"} err="failed to get container status \"820fbfdbeed78f266900f1b1796951e73c390513a8e1338d5618eefca1078777\": rpc error: code = NotFound desc = could not find container \"820fbfdbeed78f266900f1b1796951e73c390513a8e1338d5618eefca1078777\": container with ID starting with 820fbfdbeed78f266900f1b1796951e73c390513a8e1338d5618eefca1078777 not found: ID does not exist" Oct 29 23:49:00.951212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.950027 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs"] Oct 29 23:49:00.964244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.960665 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-26b322-59bc4f9d47-vzxjs"] Oct 29 23:49:00.965660 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.965629 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx"] Oct 29 23:49:00.971051 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:00.971024 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-26b322-5b7b56bd4d-w9bzx"] Oct 29 23:49:01.484285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:01.484254 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="139df2e5-deb8-44b0-8ed5-fb76650beb06" path="/var/lib/kubelet/pods/139df2e5-deb8-44b0-8ed5-fb76650beb06/volumes" Oct 29 23:49:01.484818 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:01.484801 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec6c14d-ff01-457a-abaf-8eab25418539" path="/var/lib/kubelet/pods/1ec6c14d-ff01-457a-abaf-8eab25418539/volumes" Oct 29 23:49:03.914938 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:49:03.914902 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:49:06.444844 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.444796 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" containerID="cri-o://5ab12b8f8708b4a4f9e9087744f034ee4b6c5d98083ed3f661d2acd0d2fc76de" gracePeriod=45 Oct 29 23:49:06.463636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.463600 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": dial tcp 10.132.0.85:8000: connect: connection refused" Oct 29 23:49:06.826715 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.826680 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:49:06.922653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.922614 5057 generic.go:356] "Generic (PLEG): container finished" podID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerID="5ab12b8f8708b4a4f9e9087744f034ee4b6c5d98083ed3f661d2acd0d2fc76de" exitCode=0 Oct 29 23:49:06.922858 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.922681 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" Oct 29 23:49:06.922858 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.922766 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" event={"ID":"5eee59f4-800b-4a65-a111-f11fe3afdafe","Type":"ContainerDied","Data":"5ab12b8f8708b4a4f9e9087744f034ee4b6c5d98083ed3f661d2acd0d2fc76de"} Oct 29 23:49:06.922858 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.922802 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7" event={"ID":"5eee59f4-800b-4a65-a111-f11fe3afdafe","Type":"ContainerDied","Data":"8cd158039bc8211cb306d61202a0639849fe7d526b8ff63ea76482e7c8c26541"} Oct 29 23:49:06.922858 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.922823 5057 scope.go:122] "RemoveContainer" containerID="5ab12b8f8708b4a4f9e9087744f034ee4b6c5d98083ed3f661d2acd0d2fc76de" Oct 29 23:49:06.947589 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.946492 5057 scope.go:122] "RemoveContainer" containerID="5ab12b8f8708b4a4f9e9087744f034ee4b6c5d98083ed3f661d2acd0d2fc76de" Oct 29 23:49:06.948568 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:49:06.948478 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab12b8f8708b4a4f9e9087744f034ee4b6c5d98083ed3f661d2acd0d2fc76de\": container with ID starting with 5ab12b8f8708b4a4f9e9087744f034ee4b6c5d98083ed3f661d2acd0d2fc76de not found: ID does not exist" containerID="5ab12b8f8708b4a4f9e9087744f034ee4b6c5d98083ed3f661d2acd0d2fc76de" Oct 29 23:49:06.948568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.948521 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab12b8f8708b4a4f9e9087744f034ee4b6c5d98083ed3f661d2acd0d2fc76de"} err="failed to get container status \"5ab12b8f8708b4a4f9e9087744f034ee4b6c5d98083ed3f661d2acd0d2fc76de\": rpc error: code = NotFound desc = could not find container \"5ab12b8f8708b4a4f9e9087744f034ee4b6c5d98083ed3f661d2acd0d2fc76de\": container with ID starting with 5ab12b8f8708b4a4f9e9087744f034ee4b6c5d98083ed3f661d2acd0d2fc76de not found: ID does not exist" Oct 29 23:49:06.965027 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.964986 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-tmp-dir\") pod \"5eee59f4-800b-4a65-a111-f11fe3afdafe\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " Oct 29 23:49:06.965739 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.965716 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-model-cache\") pod \"5eee59f4-800b-4a65-a111-f11fe3afdafe\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " Oct 29 23:49:06.966298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.966277 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eee59f4-800b-4a65-a111-f11fe3afdafe-tls-certs\" (UniqueName: \"kubernetes.io/secret/5eee59f4-800b-4a65-a111-f11fe3afdafe-tls-certs\") pod \"5eee59f4-800b-4a65-a111-f11fe3afdafe\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " Oct 29 23:49:06.966766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.966745 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-dshm\") pod \"5eee59f4-800b-4a65-a111-f11fe3afdafe\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " Oct 29 23:49:06.966943 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.966926 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-home\" (UniqueName: \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-home\") pod \"5eee59f4-800b-4a65-a111-f11fe3afdafe\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " Oct 29 23:49:06.967134 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.967116 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eee59f4-800b-4a65-a111-f11fe3afdafe-kube-api-access-hls9g\" (UniqueName: \"kubernetes.io/projected/5eee59f4-800b-4a65-a111-f11fe3afdafe-kube-api-access-hls9g\") pod \"5eee59f4-800b-4a65-a111-f11fe3afdafe\" (UID: \"5eee59f4-800b-4a65-a111-f11fe3afdafe\") " Oct 29 23:49:06.967406 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.965426 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-tmp-dir" pod "5eee59f4-800b-4a65-a111-f11fe3afdafe" (UID: "5eee59f4-800b-4a65-a111-f11fe3afdafe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:06.967492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.966068 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-model-cache" pod "5eee59f4-800b-4a65-a111-f11fe3afdafe" (UID: "5eee59f4-800b-4a65-a111-f11fe3afdafe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:06.967492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.967250 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-home" pod "5eee59f4-800b-4a65-a111-f11fe3afdafe" (UID: "5eee59f4-800b-4a65-a111-f11fe3afdafe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:06.968361 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.968330 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-home\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:06.968512 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.968496 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-tmp-dir\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:06.968633 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.968618 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-model-cache\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:06.972135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.972110 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-dshm" pod "5eee59f4-800b-4a65-a111-f11fe3afdafe" (UID: "5eee59f4-800b-4a65-a111-f11fe3afdafe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:06.972271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.972227 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eee59f4-800b-4a65-a111-f11fe3afdafe-tls-certs" pod "5eee59f4-800b-4a65-a111-f11fe3afdafe" (UID: "5eee59f4-800b-4a65-a111-f11fe3afdafe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:49:06.972787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:06.972751 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eee59f4-800b-4a65-a111-f11fe3afdafe-kube-api-access-hls9g" pod "5eee59f4-800b-4a65-a111-f11fe3afdafe" (UID: "5eee59f4-800b-4a65-a111-f11fe3afdafe"). InnerVolumeSpecName "kube-api-access-hls9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:49:07.069996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:07.069916 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eee59f4-800b-4a65-a111-f11fe3afdafe-tls-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:07.069996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:07.069946 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5eee59f4-800b-4a65-a111-f11fe3afdafe-dshm\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:07.069996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:07.069957 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hls9g\" (UniqueName: \"kubernetes.io/projected/5eee59f4-800b-4a65-a111-f11fe3afdafe-kube-api-access-hls9g\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:07.096311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:07.096277 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95"] Oct 29 23:49:07.281028 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:07.280993 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7"] Oct 29 23:49:07.288282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:07.288257 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e57216/distinct-w0-e57216-kserve-5f7fc89c57-nd7l7"] Oct 29 23:49:07.483339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:07.483306 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" path="/var/lib/kubelet/pods/5eee59f4-800b-4a65-a111-f11fe3afdafe/volumes" Oct 29 23:49:12.096942 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:12.096881 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95" podUID="ed5ee942-63e4-448f-b180-6790f887ffa6" containerName="payload-processing" containerID="cri-o://e0bd03ebbd7f56ca74c74dad59864fa3ce069df4080483003ab3000f87b69032" gracePeriod=25 Oct 29 23:49:17.871585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:17.871483 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm"] Oct 29 23:49:17.872060 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:17.871760 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" podUID="10e4f8ea-9a96-4cd7-875e-e9ee710c6515" containerName="maas-api" containerID="cri-o://11f8d506b93a4bcc5d437b052ef6387cb606259daabe89bdeabfec7c6b3ad647" gracePeriod=30 Oct 29 23:49:18.014926 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:18.014891 5057 generic.go:356] "Generic (PLEG): container finished" podID="10e4f8ea-9a96-4cd7-875e-e9ee710c6515" containerID="11f8d506b93a4bcc5d437b052ef6387cb606259daabe89bdeabfec7c6b3ad647" exitCode=0 Oct 29 23:49:18.015130 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:18.014972 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" event={"ID":"10e4f8ea-9a96-4cd7-875e-e9ee710c6515","Type":"ContainerDied","Data":"11f8d506b93a4bcc5d437b052ef6387cb606259daabe89bdeabfec7c6b3ad647"} Oct 29 23:49:18.392658 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:18.392625 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" Oct 29 23:49:18.541273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:18.528590 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-openshift-service-ca\") pod \"10e4f8ea-9a96-4cd7-875e-e9ee710c6515\" (UID: \"10e4f8ea-9a96-4cd7-875e-e9ee710c6515\") " Oct 29 23:49:18.541273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:18.528768 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-kube-api-access-6nn2k\" (UniqueName: \"kubernetes.io/projected/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-kube-api-access-6nn2k\") pod \"10e4f8ea-9a96-4cd7-875e-e9ee710c6515\" (UID: \"10e4f8ea-9a96-4cd7-875e-e9ee710c6515\") " Oct 29 23:49:18.541273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:18.528817 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-metrics-certs\") pod \"10e4f8ea-9a96-4cd7-875e-e9ee710c6515\" (UID: \"10e4f8ea-9a96-4cd7-875e-e9ee710c6515\") " Oct 29 23:49:18.541273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:18.528906 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-maas-api-tls\") pod \"10e4f8ea-9a96-4cd7-875e-e9ee710c6515\" (UID: \"10e4f8ea-9a96-4cd7-875e-e9ee710c6515\") " Oct 29 23:49:18.543864 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:18.543830 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-openshift-service-ca" pod "10e4f8ea-9a96-4cd7-875e-e9ee710c6515" (UID: "10e4f8ea-9a96-4cd7-875e-e9ee710c6515"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:49:18.580843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:18.580791 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-kube-api-access-6nn2k" pod "10e4f8ea-9a96-4cd7-875e-e9ee710c6515" (UID: "10e4f8ea-9a96-4cd7-875e-e9ee710c6515"). InnerVolumeSpecName "kube-api-access-6nn2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:49:18.581216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:18.581163 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-maas-api-tls" pod "10e4f8ea-9a96-4cd7-875e-e9ee710c6515" (UID: "10e4f8ea-9a96-4cd7-875e-e9ee710c6515"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:49:18.587689 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:18.587661 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-metrics-certs" pod "10e4f8ea-9a96-4cd7-875e-e9ee710c6515" (UID: "10e4f8ea-9a96-4cd7-875e-e9ee710c6515"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:49:18.651028 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:18.640889 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:18.651028 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:18.640927 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nn2k\" (UniqueName: \"kubernetes.io/projected/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-kube-api-access-6nn2k\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:18.651028 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:18.640942 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:18.651028 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:18.640955 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10e4f8ea-9a96-4cd7-875e-e9ee710c6515-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:18.758982 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:18.758938 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:49:19.082646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:19.082607 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" event={"ID":"10e4f8ea-9a96-4cd7-875e-e9ee710c6515","Type":"ContainerDied","Data":"88df9e46e4d881642b687e256425f7316b7e40cb274f662007dd3c244f6e43b2"} Oct 29 23:49:19.083267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:19.083242 5057 scope.go:122] "RemoveContainer" containerID="11f8d506b93a4bcc5d437b052ef6387cb606259daabe89bdeabfec7c6b3ad647" Oct 29 23:49:19.083619 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:19.083596 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm" Oct 29 23:49:19.171302 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:19.171161 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm"] Oct 29 23:49:19.176489 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:19.176458 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-26b322-84c555554b-pd6gm"] Oct 29 23:49:19.485837 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:19.485804 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10e4f8ea-9a96-4cd7-875e-e9ee710c6515" path="/var/lib/kubelet/pods/10e4f8ea-9a96-4cd7-875e-e9ee710c6515/volumes" Oct 29 23:49:20.375538 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:20.375286 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95" podUID="ed5ee942-63e4-448f-b180-6790f887ffa6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:49:27.731335 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:27.731303 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh"] Oct 29 23:49:27.731848 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:27.731525 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" podUID="d89378ad-ddab-4899-9041-7c4cabafa109" containerName="istio-proxy" containerID="cri-o://4b0869b5d5ac71822fff422c7c14f9e1f26b1f3369a4beefde36cf8eef225651" gracePeriod=30 Oct 29 23:49:30.399216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:30.398814 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95" podUID="ed5ee942-63e4-448f-b180-6790f887ffa6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:49:32.219302 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:32.219271 5057 generic.go:356] "Generic (PLEG): container finished" podID="ed5ee942-63e4-448f-b180-6790f887ffa6" containerID="e0bd03ebbd7f56ca74c74dad59864fa3ce069df4080483003ab3000f87b69032" exitCode=0 Oct 29 23:49:32.219614 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:32.219342 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95" event={"ID":"ed5ee942-63e4-448f-b180-6790f887ffa6","Type":"ContainerDied","Data":"e0bd03ebbd7f56ca74c74dad59864fa3ce069df4080483003ab3000f87b69032"} Oct 29 23:49:32.443125 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:32.443090 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95" Oct 29 23:49:32.461251 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:32.461224 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed5ee942-63e4-448f-b180-6790f887ffa6-kube-api-access-86rf8\" (UniqueName: \"kubernetes.io/projected/ed5ee942-63e4-448f-b180-6790f887ffa6-kube-api-access-86rf8\") pod \"ed5ee942-63e4-448f-b180-6790f887ffa6\" (UID: \"ed5ee942-63e4-448f-b180-6790f887ffa6\") " Oct 29 23:49:32.461431 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:32.461267 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed5ee942-63e4-448f-b180-6790f887ffa6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed5ee942-63e4-448f-b180-6790f887ffa6-plugins-config-volume\") pod \"ed5ee942-63e4-448f-b180-6790f887ffa6\" (UID: \"ed5ee942-63e4-448f-b180-6790f887ffa6\") " Oct 29 23:49:32.461752 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:32.461717 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed5ee942-63e4-448f-b180-6790f887ffa6-plugins-config-volume" pod "ed5ee942-63e4-448f-b180-6790f887ffa6" (UID: "ed5ee942-63e4-448f-b180-6790f887ffa6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:49:32.462061 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:32.462042 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed5ee942-63e4-448f-b180-6790f887ffa6-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:32.463948 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:32.463919 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5ee942-63e4-448f-b180-6790f887ffa6-kube-api-access-86rf8" pod "ed5ee942-63e4-448f-b180-6790f887ffa6" (UID: "ed5ee942-63e4-448f-b180-6790f887ffa6"). InnerVolumeSpecName "kube-api-access-86rf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:49:32.583210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:32.564317 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86rf8\" (UniqueName: \"kubernetes.io/projected/ed5ee942-63e4-448f-b180-6790f887ffa6-kube-api-access-86rf8\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:33.008512 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.008483 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:49:33.072232 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.072179 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-istio-data\") pod \"d89378ad-ddab-4899-9041-7c4cabafa109\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " Oct 29 23:49:33.072448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.072253 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89378ad-ddab-4899-9041-7c4cabafa109-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d89378ad-ddab-4899-9041-7c4cabafa109-istiod-ca-cert\") pod \"d89378ad-ddab-4899-9041-7c4cabafa109\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " Oct 29 23:49:33.072448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.072362 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-workload-socket\") pod \"d89378ad-ddab-4899-9041-7c4cabafa109\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " Oct 29 23:49:33.072583 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.072450 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-istio-data" pod "d89378ad-ddab-4899-9041-7c4cabafa109" (UID: "d89378ad-ddab-4899-9041-7c4cabafa109"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:33.072583 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.072483 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d89378ad-ddab-4899-9041-7c4cabafa109-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d89378ad-ddab-4899-9041-7c4cabafa109-istio-podinfo\") pod \"d89378ad-ddab-4899-9041-7c4cabafa109\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " Oct 29 23:49:33.072583 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.072566 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-credential-socket\") pod \"d89378ad-ddab-4899-9041-7c4cabafa109\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " Oct 29 23:49:33.072703 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.072662 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89378ad-ddab-4899-9041-7c4cabafa109-kube-api-access-mvp4w\" (UniqueName: \"kubernetes.io/projected/d89378ad-ddab-4899-9041-7c4cabafa109-kube-api-access-mvp4w\") pod \"d89378ad-ddab-4899-9041-7c4cabafa109\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " Oct 29 23:49:33.072703 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.072664 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-workload-socket" pod "d89378ad-ddab-4899-9041-7c4cabafa109" (UID: "d89378ad-ddab-4899-9041-7c4cabafa109"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:33.072703 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.072677 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89378ad-ddab-4899-9041-7c4cabafa109-istiod-ca-cert" pod "d89378ad-ddab-4899-9041-7c4cabafa109" (UID: "d89378ad-ddab-4899-9041-7c4cabafa109"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:49:33.072878 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.072716 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89378ad-ddab-4899-9041-7c4cabafa109-istio-token\" (UniqueName: \"kubernetes.io/projected/d89378ad-ddab-4899-9041-7c4cabafa109-istio-token\") pod \"d89378ad-ddab-4899-9041-7c4cabafa109\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " Oct 29 23:49:33.072878 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.072778 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-workload-certs\") pod \"d89378ad-ddab-4899-9041-7c4cabafa109\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " Oct 29 23:49:33.072878 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.072801 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-istio-envoy\") pod \"d89378ad-ddab-4899-9041-7c4cabafa109\" (UID: \"d89378ad-ddab-4899-9041-7c4cabafa109\") " Oct 29 23:49:33.072878 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.072820 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-credential-socket" pod "d89378ad-ddab-4899-9041-7c4cabafa109" (UID: "d89378ad-ddab-4899-9041-7c4cabafa109"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:33.073093 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.072988 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-workload-certs" pod "d89378ad-ddab-4899-9041-7c4cabafa109" (UID: "d89378ad-ddab-4899-9041-7c4cabafa109"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:33.073529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.073504 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:33.073659 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.073534 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:33.073659 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.073551 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:33.073659 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.073564 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:33.073659 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.073578 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d89378ad-ddab-4899-9041-7c4cabafa109-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:33.074682 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.074656 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d89378ad-ddab-4899-9041-7c4cabafa109-istio-podinfo" pod "d89378ad-ddab-4899-9041-7c4cabafa109" (UID: "d89378ad-ddab-4899-9041-7c4cabafa109"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:49:33.074944 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.074904 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89378ad-ddab-4899-9041-7c4cabafa109-istio-token" pod "d89378ad-ddab-4899-9041-7c4cabafa109" (UID: "d89378ad-ddab-4899-9041-7c4cabafa109"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:49:33.074944 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.074919 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-istio-envoy" pod "d89378ad-ddab-4899-9041-7c4cabafa109" (UID: "d89378ad-ddab-4899-9041-7c4cabafa109"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:33.074944 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.074932 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89378ad-ddab-4899-9041-7c4cabafa109-kube-api-access-mvp4w" pod "d89378ad-ddab-4899-9041-7c4cabafa109" (UID: "d89378ad-ddab-4899-9041-7c4cabafa109"). InnerVolumeSpecName "kube-api-access-mvp4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:49:33.174948 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.174910 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d89378ad-ddab-4899-9041-7c4cabafa109-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:33.174948 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.174944 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvp4w\" (UniqueName: \"kubernetes.io/projected/d89378ad-ddab-4899-9041-7c4cabafa109-kube-api-access-mvp4w\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:33.175209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.174959 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d89378ad-ddab-4899-9041-7c4cabafa109-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:33.175209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.174971 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d89378ad-ddab-4899-9041-7c4cabafa109-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:33.230551 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.230521 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95" Oct 29 23:49:33.230998 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.230562 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95" event={"ID":"ed5ee942-63e4-448f-b180-6790f887ffa6","Type":"ContainerDied","Data":"d183e9ea1c8db2a71da46e621728cce1ddfb6a2c38eb96dbf773649300691c77"} Oct 29 23:49:33.230998 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.230637 5057 scope.go:122] "RemoveContainer" containerID="e0bd03ebbd7f56ca74c74dad59864fa3ce069df4080483003ab3000f87b69032" Oct 29 23:49:33.232939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.232903 5057 generic.go:356] "Generic (PLEG): container finished" podID="d89378ad-ddab-4899-9041-7c4cabafa109" containerID="4b0869b5d5ac71822fff422c7c14f9e1f26b1f3369a4beefde36cf8eef225651" exitCode=0 Oct 29 23:49:33.233073 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.232947 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" event={"ID":"d89378ad-ddab-4899-9041-7c4cabafa109","Type":"ContainerDied","Data":"4b0869b5d5ac71822fff422c7c14f9e1f26b1f3369a4beefde36cf8eef225651"} Oct 29 23:49:33.233073 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.232961 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" Oct 29 23:49:33.233073 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.232986 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh" event={"ID":"d89378ad-ddab-4899-9041-7c4cabafa109","Type":"ContainerDied","Data":"39bd9e81c0d191f8de486c0e62854923b9472a1f2c92c4473003244a0c9b36d6"} Oct 29 23:49:33.243866 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.243844 5057 scope.go:122] "RemoveContainer" containerID="4b0869b5d5ac71822fff422c7c14f9e1f26b1f3369a4beefde36cf8eef225651" Oct 29 23:49:33.255673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.255652 5057 scope.go:122] "RemoveContainer" containerID="38f1f62e9e3c5f520f43a26031fde07bebe829a1260e36b8ac12b46fb3d650d0" Oct 29 23:49:33.266929 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.266908 5057 scope.go:122] "RemoveContainer" containerID="4b0869b5d5ac71822fff422c7c14f9e1f26b1f3369a4beefde36cf8eef225651" Oct 29 23:49:33.267276 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:49:33.267251 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b0869b5d5ac71822fff422c7c14f9e1f26b1f3369a4beefde36cf8eef225651\": container with ID starting with 4b0869b5d5ac71822fff422c7c14f9e1f26b1f3369a4beefde36cf8eef225651 not found: ID does not exist" containerID="4b0869b5d5ac71822fff422c7c14f9e1f26b1f3369a4beefde36cf8eef225651" Oct 29 23:49:33.267379 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.267287 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b0869b5d5ac71822fff422c7c14f9e1f26b1f3369a4beefde36cf8eef225651"} err="failed to get container status \"4b0869b5d5ac71822fff422c7c14f9e1f26b1f3369a4beefde36cf8eef225651\": rpc error: code = NotFound desc = could not find container \"4b0869b5d5ac71822fff422c7c14f9e1f26b1f3369a4beefde36cf8eef225651\": container with ID starting with 4b0869b5d5ac71822fff422c7c14f9e1f26b1f3369a4beefde36cf8eef225651 not found: ID does not exist" Oct 29 23:49:33.267379 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.267311 5057 scope.go:122] "RemoveContainer" containerID="38f1f62e9e3c5f520f43a26031fde07bebe829a1260e36b8ac12b46fb3d650d0" Oct 29 23:49:33.267584 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:49:33.267560 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f1f62e9e3c5f520f43a26031fde07bebe829a1260e36b8ac12b46fb3d650d0\": container with ID starting with 38f1f62e9e3c5f520f43a26031fde07bebe829a1260e36b8ac12b46fb3d650d0 not found: ID does not exist" containerID="38f1f62e9e3c5f520f43a26031fde07bebe829a1260e36b8ac12b46fb3d650d0" Oct 29 23:49:33.267645 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.267594 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f1f62e9e3c5f520f43a26031fde07bebe829a1260e36b8ac12b46fb3d650d0"} err="failed to get container status \"38f1f62e9e3c5f520f43a26031fde07bebe829a1260e36b8ac12b46fb3d650d0\": rpc error: code = NotFound desc = could not find container \"38f1f62e9e3c5f520f43a26031fde07bebe829a1260e36b8ac12b46fb3d650d0\": container with ID starting with 38f1f62e9e3c5f520f43a26031fde07bebe829a1260e36b8ac12b46fb3d650d0 not found: ID does not exist" Oct 29 23:49:33.319718 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.319668 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh"] Oct 29 23:49:33.326226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.326177 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-26b322-openshift-default-94cbc5f57-dtfvh"] Oct 29 23:49:33.388414 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.388382 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95"] Oct 29 23:49:33.407961 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.407929 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e57216-6f8c68c6db-x7f95"] Oct 29 23:49:33.484353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.484313 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d89378ad-ddab-4899-9041-7c4cabafa109" path="/var/lib/kubelet/pods/d89378ad-ddab-4899-9041-7c4cabafa109/volumes" Oct 29 23:49:33.484785 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:33.484769 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed5ee942-63e4-448f-b180-6790f887ffa6" path="/var/lib/kubelet/pods/ed5ee942-63e4-448f-b180-6790f887ffa6/volumes" Oct 29 23:49:34.326277 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:49:34.320385 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:49:38.907303 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:38.907257 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6"] Oct 29 23:49:43.908424 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:43.908314 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6" podUID="6225f8f9-8cbb-46cb-9563-311fab3d5e21" containerName="payload-pre-processing" containerID="cri-o://ac83edbf62004f98d5dbaba36a936d9b5a6507c54af824996d27f14c4553cf47" gracePeriod=25 Oct 29 23:49:44.228615 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:44.228583 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6" Oct 29 23:49:44.316413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:44.316378 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6225f8f9-8cbb-46cb-9563-311fab3d5e21-kube-api-access-787cv\" (UniqueName: \"kubernetes.io/projected/6225f8f9-8cbb-46cb-9563-311fab3d5e21-kube-api-access-787cv\") pod \"6225f8f9-8cbb-46cb-9563-311fab3d5e21\" (UID: \"6225f8f9-8cbb-46cb-9563-311fab3d5e21\") " Oct 29 23:49:44.317633 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:44.316760 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6225f8f9-8cbb-46cb-9563-311fab3d5e21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6225f8f9-8cbb-46cb-9563-311fab3d5e21-plugins-config-volume\") pod \"6225f8f9-8cbb-46cb-9563-311fab3d5e21\" (UID: \"6225f8f9-8cbb-46cb-9563-311fab3d5e21\") " Oct 29 23:49:44.317633 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:44.317148 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6225f8f9-8cbb-46cb-9563-311fab3d5e21-plugins-config-volume" pod "6225f8f9-8cbb-46cb-9563-311fab3d5e21" (UID: "6225f8f9-8cbb-46cb-9563-311fab3d5e21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:49:44.317633 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:44.317595 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6225f8f9-8cbb-46cb-9563-311fab3d5e21-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:44.326435 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:44.326399 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6225f8f9-8cbb-46cb-9563-311fab3d5e21-kube-api-access-787cv" pod "6225f8f9-8cbb-46cb-9563-311fab3d5e21" (UID: "6225f8f9-8cbb-46cb-9563-311fab3d5e21"). InnerVolumeSpecName "kube-api-access-787cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:49:44.419678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:44.419647 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-787cv\" (UniqueName: \"kubernetes.io/projected/6225f8f9-8cbb-46cb-9563-311fab3d5e21-kube-api-access-787cv\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:49:44.458647 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:44.458552 5057 generic.go:356] "Generic (PLEG): container finished" podID="6225f8f9-8cbb-46cb-9563-311fab3d5e21" containerID="ac83edbf62004f98d5dbaba36a936d9b5a6507c54af824996d27f14c4553cf47" exitCode=0 Oct 29 23:49:44.458647 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:44.458608 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6" Oct 29 23:49:44.458903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:44.458644 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6" event={"ID":"6225f8f9-8cbb-46cb-9563-311fab3d5e21","Type":"ContainerDied","Data":"ac83edbf62004f98d5dbaba36a936d9b5a6507c54af824996d27f14c4553cf47"} Oct 29 23:49:44.458903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:44.458711 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6" event={"ID":"6225f8f9-8cbb-46cb-9563-311fab3d5e21","Type":"ContainerDied","Data":"884de911b0e399207d48302396c86ae31c55667b72df2da801267dc2394ee33b"} Oct 29 23:49:44.458903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:44.458743 5057 scope.go:122] "RemoveContainer" containerID="ac83edbf62004f98d5dbaba36a936d9b5a6507c54af824996d27f14c4553cf47" Oct 29 23:49:44.471706 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:44.471686 5057 scope.go:122] "RemoveContainer" containerID="ac83edbf62004f98d5dbaba36a936d9b5a6507c54af824996d27f14c4553cf47" Oct 29 23:49:44.471946 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:49:44.471928 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac83edbf62004f98d5dbaba36a936d9b5a6507c54af824996d27f14c4553cf47\": container with ID starting with ac83edbf62004f98d5dbaba36a936d9b5a6507c54af824996d27f14c4553cf47 not found: ID does not exist" containerID="ac83edbf62004f98d5dbaba36a936d9b5a6507c54af824996d27f14c4553cf47" Oct 29 23:49:44.471989 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:44.471955 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac83edbf62004f98d5dbaba36a936d9b5a6507c54af824996d27f14c4553cf47"} err="failed to get container status \"ac83edbf62004f98d5dbaba36a936d9b5a6507c54af824996d27f14c4553cf47\": rpc error: code = NotFound desc = could not find container \"ac83edbf62004f98d5dbaba36a936d9b5a6507c54af824996d27f14c4553cf47\": container with ID starting with ac83edbf62004f98d5dbaba36a936d9b5a6507c54af824996d27f14c4553cf47 not found: ID does not exist" Oct 29 23:49:44.536700 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:44.536669 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6"] Oct 29 23:49:44.541392 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:44.541368 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2591f6-59d9d5cc79-z2mj6"] Oct 29 23:49:45.482894 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:49:45.482861 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6225f8f9-8cbb-46cb-9563-311fab3d5e21" path="/var/lib/kubelet/pods/6225f8f9-8cbb-46cb-9563-311fab3d5e21/volumes" Oct 29 23:50:03.916604 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:50:03.916565 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:50:04.214572 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:04.214541 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:50:06.266578 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:06.266541 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5"] Oct 29 23:50:06.274542 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:06.274502 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" podUID="772cd9de-1148-400d-9c3b-f0fb52455b8b" containerName="maas-api" containerID="cri-o://c7c87fb5cef0149f8c1aa7cbe4076eef7708a2add63fc72542bf1dfe4b5313f1" gracePeriod=30 Oct 29 23:50:06.668602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:06.668511 5057 generic.go:356] "Generic (PLEG): container finished" podID="772cd9de-1148-400d-9c3b-f0fb52455b8b" containerID="c7c87fb5cef0149f8c1aa7cbe4076eef7708a2add63fc72542bf1dfe4b5313f1" exitCode=0 Oct 29 23:50:06.669112 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:06.668839 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" event={"ID":"772cd9de-1148-400d-9c3b-f0fb52455b8b","Type":"ContainerDied","Data":"c7c87fb5cef0149f8c1aa7cbe4076eef7708a2add63fc72542bf1dfe4b5313f1"} Oct 29 23:50:06.836305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:06.836265 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" Oct 29 23:50:06.945560 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:06.945527 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/772cd9de-1148-400d-9c3b-f0fb52455b8b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772cd9de-1148-400d-9c3b-f0fb52455b8b-openshift-service-ca\") pod \"772cd9de-1148-400d-9c3b-f0fb52455b8b\" (UID: \"772cd9de-1148-400d-9c3b-f0fb52455b8b\") " Oct 29 23:50:06.945774 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:06.945586 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/772cd9de-1148-400d-9c3b-f0fb52455b8b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772cd9de-1148-400d-9c3b-f0fb52455b8b-maas-api-tls\") pod \"772cd9de-1148-400d-9c3b-f0fb52455b8b\" (UID: \"772cd9de-1148-400d-9c3b-f0fb52455b8b\") " Oct 29 23:50:06.945850 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:06.945836 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/772cd9de-1148-400d-9c3b-f0fb52455b8b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/772cd9de-1148-400d-9c3b-f0fb52455b8b-metrics-certs\") pod \"772cd9de-1148-400d-9c3b-f0fb52455b8b\" (UID: \"772cd9de-1148-400d-9c3b-f0fb52455b8b\") " Oct 29 23:50:06.945949 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:06.945932 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772cd9de-1148-400d-9c3b-f0fb52455b8b-kube-api-access-hjq24\" (UniqueName: \"kubernetes.io/projected/772cd9de-1148-400d-9c3b-f0fb52455b8b-kube-api-access-hjq24\") pod \"772cd9de-1148-400d-9c3b-f0fb52455b8b\" (UID: \"772cd9de-1148-400d-9c3b-f0fb52455b8b\") " Oct 29 23:50:06.946396 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:06.946352 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772cd9de-1148-400d-9c3b-f0fb52455b8b-openshift-service-ca" pod "772cd9de-1148-400d-9c3b-f0fb52455b8b" (UID: "772cd9de-1148-400d-9c3b-f0fb52455b8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:50:06.946827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:06.946803 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772cd9de-1148-400d-9c3b-f0fb52455b8b-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:06.948156 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:06.948124 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772cd9de-1148-400d-9c3b-f0fb52455b8b-metrics-certs" pod "772cd9de-1148-400d-9c3b-f0fb52455b8b" (UID: "772cd9de-1148-400d-9c3b-f0fb52455b8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:50:06.948465 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:06.948349 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772cd9de-1148-400d-9c3b-f0fb52455b8b-maas-api-tls" pod "772cd9de-1148-400d-9c3b-f0fb52455b8b" (UID: "772cd9de-1148-400d-9c3b-f0fb52455b8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:50:06.948544 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:06.948502 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772cd9de-1148-400d-9c3b-f0fb52455b8b-kube-api-access-hjq24" pod "772cd9de-1148-400d-9c3b-f0fb52455b8b" (UID: "772cd9de-1148-400d-9c3b-f0fb52455b8b"). InnerVolumeSpecName "kube-api-access-hjq24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:07.049261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:07.049168 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjq24\" (UniqueName: \"kubernetes.io/projected/772cd9de-1148-400d-9c3b-f0fb52455b8b-kube-api-access-hjq24\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:07.049261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:07.049222 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772cd9de-1148-400d-9c3b-f0fb52455b8b-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:07.049261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:07.049237 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772cd9de-1148-400d-9c3b-f0fb52455b8b-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:07.678318 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:07.678288 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" Oct 29 23:50:07.678814 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:07.678317 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5" event={"ID":"772cd9de-1148-400d-9c3b-f0fb52455b8b","Type":"ContainerDied","Data":"9b32ceedf71ced0dd17a66d29d9722d7399b664b9cd0dab83d73a67e93fec4b5"} Oct 29 23:50:07.678814 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:07.678394 5057 scope.go:122] "RemoveContainer" containerID="c7c87fb5cef0149f8c1aa7cbe4076eef7708a2add63fc72542bf1dfe4b5313f1" Oct 29 23:50:07.733248 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:07.733202 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5"] Oct 29 23:50:07.744019 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:07.743990 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed8218-594ff6d6d9-mlfp5"] Oct 29 23:50:09.484046 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:09.484014 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772cd9de-1148-400d-9c3b-f0fb52455b8b" path="/var/lib/kubelet/pods/772cd9de-1148-400d-9c3b-f0fb52455b8b/volumes" Oct 29 23:50:14.361947 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:14.361902 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq"] Oct 29 23:50:14.362510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:14.362151 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" podUID="ea5a1a45-f903-4dcb-8d39-fabd9bf226e4" containerName="istio-proxy" containerID="cri-o://bd9108463c34b48f8ec48e2128b8955fc67f20ada8e4809004bc00c5a4e5b3f8" gracePeriod=30 Oct 29 23:50:19.780866 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.780831 5057 generic.go:356] "Generic (PLEG): container finished" podID="ea5a1a45-f903-4dcb-8d39-fabd9bf226e4" containerID="bd9108463c34b48f8ec48e2128b8955fc67f20ada8e4809004bc00c5a4e5b3f8" exitCode=0 Oct 29 23:50:19.782934 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.780904 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" event={"ID":"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4","Type":"ContainerDied","Data":"bd9108463c34b48f8ec48e2128b8955fc67f20ada8e4809004bc00c5a4e5b3f8"} Oct 29 23:50:19.782934 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.782880 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" event={"ID":"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4","Type":"ContainerDied","Data":"18f20be1ac6c1986864551d2b8b4e4bcabd60e20b8a59d9817e6a6ada0d9cdae"} Oct 29 23:50:19.782934 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.782913 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18f20be1ac6c1986864551d2b8b4e4bcabd60e20b8a59d9817e6a6ada0d9cdae" Oct 29 23:50:19.801075 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.801054 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 23:50:19.883363 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.883328 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-workload-socket\") pod \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " Oct 29 23:50:19.883568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.883469 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-token\" (UniqueName: \"kubernetes.io/projected/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-token\") pod \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " Oct 29 23:50:19.883568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.883543 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-workload-certs\") pod \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " Oct 29 23:50:19.883713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.883582 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-kube-api-access-7w2pm\" (UniqueName: \"kubernetes.io/projected/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-kube-api-access-7w2pm\") pod \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " Oct 29 23:50:19.883713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.883611 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-data\") pod \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " Oct 29 23:50:19.883713 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.883628 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-workload-socket" pod "ea5a1a45-f903-4dcb-8d39-fabd9bf226e4" (UID: "ea5a1a45-f903-4dcb-8d39-fabd9bf226e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:19.883896 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.883753 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istiod-ca-cert\") pod \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " Oct 29 23:50:19.883896 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.883818 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-credential-socket\") pod \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " Oct 29 23:50:19.883896 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.883822 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-workload-certs" pod "ea5a1a45-f903-4dcb-8d39-fabd9bf226e4" (UID: "ea5a1a45-f903-4dcb-8d39-fabd9bf226e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:19.884070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.883948 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-podinfo\") pod \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " Oct 29 23:50:19.884070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.883980 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-data" pod "ea5a1a45-f903-4dcb-8d39-fabd9bf226e4" (UID: "ea5a1a45-f903-4dcb-8d39-fabd9bf226e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:19.884070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.883991 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-envoy\") pod \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\" (UID: \"ea5a1a45-f903-4dcb-8d39-fabd9bf226e4\") " Oct 29 23:50:19.884309 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.884158 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-credential-socket" pod "ea5a1a45-f903-4dcb-8d39-fabd9bf226e4" (UID: "ea5a1a45-f903-4dcb-8d39-fabd9bf226e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:19.884449 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.884429 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istiod-ca-cert" pod "ea5a1a45-f903-4dcb-8d39-fabd9bf226e4" (UID: "ea5a1a45-f903-4dcb-8d39-fabd9bf226e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:50:19.884760 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.884737 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:19.884760 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.884764 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:19.884946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.884777 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:19.884946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.884789 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:19.884946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.884801 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:19.887266 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.887238 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-podinfo" pod "ea5a1a45-f903-4dcb-8d39-fabd9bf226e4" (UID: "ea5a1a45-f903-4dcb-8d39-fabd9bf226e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:50:19.887382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.887259 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-envoy" pod "ea5a1a45-f903-4dcb-8d39-fabd9bf226e4" (UID: "ea5a1a45-f903-4dcb-8d39-fabd9bf226e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:19.888757 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.888733 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-token" pod "ea5a1a45-f903-4dcb-8d39-fabd9bf226e4" (UID: "ea5a1a45-f903-4dcb-8d39-fabd9bf226e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:19.890356 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.890331 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-kube-api-access-7w2pm" pod "ea5a1a45-f903-4dcb-8d39-fabd9bf226e4" (UID: "ea5a1a45-f903-4dcb-8d39-fabd9bf226e4"). InnerVolumeSpecName "kube-api-access-7w2pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:19.986176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.986146 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:19.986347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.986202 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:19.986347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.986212 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:19.986347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:19.986222 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w2pm\" (UniqueName: \"kubernetes.io/projected/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4-kube-api-access-7w2pm\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:20.793128 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:20.793094 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq" Oct 29 23:50:20.861666 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:20.861629 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq"] Oct 29 23:50:20.870446 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:20.870414 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-tf4xq"] Oct 29 23:50:21.482290 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:21.482259 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea5a1a45-f903-4dcb-8d39-fabd9bf226e4" path="/var/lib/kubelet/pods/ea5a1a45-f903-4dcb-8d39-fabd9bf226e4/volumes" Oct 29 23:50:29.756324 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:29.756289 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw"] Oct 29 23:50:29.756844 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:29.756516 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" podUID="2bf37ad5-2491-488a-9cea-5f0a3872f5fb" containerName="istio-proxy" containerID="cri-o://48261d81e8d198bc853f0369345494e3611da5dd69f97a79e36e0637ac1fb174" gracePeriod=30 Oct 29 23:50:29.768750 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:29.768722 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j"] Oct 29 23:50:29.768945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:29.768910 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" podUID="f97576f7-0dd7-4515-9774-8e8849cc2c5e" containerName="istio-proxy" containerID="cri-o://e84bfaeb37855ca31279d661e919c38f573366c6cb038f3f0577484fda345a24" gracePeriod=30 Oct 29 23:50:30.275398 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:30.275362 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j"] Oct 29 23:50:33.914343 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:50:33.914305 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:50:34.958787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:34.958530 5057 generic.go:356] "Generic (PLEG): container finished" podID="f97576f7-0dd7-4515-9774-8e8849cc2c5e" containerID="e84bfaeb37855ca31279d661e919c38f573366c6cb038f3f0577484fda345a24" exitCode=0 Oct 29 23:50:34.958787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:34.958702 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" event={"ID":"f97576f7-0dd7-4515-9774-8e8849cc2c5e","Type":"ContainerDied","Data":"e84bfaeb37855ca31279d661e919c38f573366c6cb038f3f0577484fda345a24"} Oct 29 23:50:34.962343 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:34.962133 5057 generic.go:356] "Generic (PLEG): container finished" podID="2bf37ad5-2491-488a-9cea-5f0a3872f5fb" containerID="48261d81e8d198bc853f0369345494e3611da5dd69f97a79e36e0637ac1fb174" exitCode=0 Oct 29 23:50:34.962343 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:34.962279 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" event={"ID":"2bf37ad5-2491-488a-9cea-5f0a3872f5fb","Type":"ContainerDied","Data":"48261d81e8d198bc853f0369345494e3611da5dd69f97a79e36e0637ac1fb174"} Oct 29 23:50:35.095512 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.095477 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:50:35.181533 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.181290 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:50:35.209316 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.209280 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-workload-socket\") pod \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " Oct 29 23:50:35.209519 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.209332 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-kube-api-access-9ghvk\" (UniqueName: \"kubernetes.io/projected/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-kube-api-access-9ghvk\") pod \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " Oct 29 23:50:35.209519 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.209425 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-token\" (UniqueName: \"kubernetes.io/projected/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-token\") pod \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " Oct 29 23:50:35.209519 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.209487 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f97576f7-0dd7-4515-9774-8e8849cc2c5e-kube-api-access-xb5cr\" (UniqueName: \"kubernetes.io/projected/f97576f7-0dd7-4515-9774-8e8849cc2c5e-kube-api-access-xb5cr\") pod \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " Oct 29 23:50:35.209711 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.209526 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istiod-ca-cert\") pod \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " Oct 29 23:50:35.209711 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.209557 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-data\") pod \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " Oct 29 23:50:35.209711 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.209589 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-token\" (UniqueName: \"kubernetes.io/projected/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-token\") pod \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " Oct 29 23:50:35.209711 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.209615 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-envoy\") pod \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " Oct 29 23:50:35.209711 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.209609 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-workload-socket" pod "2bf37ad5-2491-488a-9cea-5f0a3872f5fb" (UID: "2bf37ad5-2491-488a-9cea-5f0a3872f5fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:35.210019 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.209981 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-data" pod "2bf37ad5-2491-488a-9cea-5f0a3872f5fb" (UID: "2bf37ad5-2491-488a-9cea-5f0a3872f5fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:35.210162 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.210136 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istiod-ca-cert\") pod \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " Oct 29 23:50:35.210319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.210234 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-workload-certs\") pod \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " Oct 29 23:50:35.210319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.210266 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-data\") pod \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " Oct 29 23:50:35.210463 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.210322 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-workload-socket\") pod \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " Oct 29 23:50:35.210463 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.210445 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-envoy\") pod \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " Oct 29 23:50:35.210602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.210478 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istiod-ca-cert" pod "f97576f7-0dd7-4515-9774-8e8849cc2c5e" (UID: "f97576f7-0dd7-4515-9774-8e8849cc2c5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:50:35.210602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.210493 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-podinfo\") pod \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " Oct 29 23:50:35.210602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.210563 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-credential-socket\") pod \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\" (UID: \"2bf37ad5-2491-488a-9cea-5f0a3872f5fb\") " Oct 29 23:50:35.210786 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.210642 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-credential-socket\") pod \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " Oct 29 23:50:35.210786 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.210670 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-workload-certs\") pod \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " Oct 29 23:50:35.210786 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.210728 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-podinfo\") pod \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\" (UID: \"f97576f7-0dd7-4515-9774-8e8849cc2c5e\") " Oct 29 23:50:35.210957 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.210928 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-workload-certs" pod "2bf37ad5-2491-488a-9cea-5f0a3872f5fb" (UID: "2bf37ad5-2491-488a-9cea-5f0a3872f5fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:35.211027 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.210990 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-data" pod "f97576f7-0dd7-4515-9774-8e8849cc2c5e" (UID: "f97576f7-0dd7-4515-9774-8e8849cc2c5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:35.211782 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.211504 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-credential-socket" pod "f97576f7-0dd7-4515-9774-8e8849cc2c5e" (UID: "f97576f7-0dd7-4515-9774-8e8849cc2c5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:35.211782 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.211589 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-workload-certs" pod "f97576f7-0dd7-4515-9774-8e8849cc2c5e" (UID: "f97576f7-0dd7-4515-9774-8e8849cc2c5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:35.211782 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.211648 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-credential-socket" pod "2bf37ad5-2491-488a-9cea-5f0a3872f5fb" (UID: "2bf37ad5-2491-488a-9cea-5f0a3872f5fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:35.211782 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.211698 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-workload-socket" pod "f97576f7-0dd7-4515-9774-8e8849cc2c5e" (UID: "f97576f7-0dd7-4515-9774-8e8849cc2c5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:35.211782 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.211756 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:35.211782 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.211776 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:35.212279 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.211790 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:35.212279 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.211804 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:35.212279 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.211817 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:35.212279 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.211831 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:35.212279 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.211946 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istiod-ca-cert" pod "2bf37ad5-2491-488a-9cea-5f0a3872f5fb" (UID: "2bf37ad5-2491-488a-9cea-5f0a3872f5fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:50:35.213146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.212548 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-kube-api-access-9ghvk" pod "2bf37ad5-2491-488a-9cea-5f0a3872f5fb" (UID: "2bf37ad5-2491-488a-9cea-5f0a3872f5fb"). InnerVolumeSpecName "kube-api-access-9ghvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:35.213418 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.213393 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-envoy" pod "f97576f7-0dd7-4515-9774-8e8849cc2c5e" (UID: "f97576f7-0dd7-4515-9774-8e8849cc2c5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:35.214231 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.214169 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-token" pod "f97576f7-0dd7-4515-9774-8e8849cc2c5e" (UID: "f97576f7-0dd7-4515-9774-8e8849cc2c5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:35.215276 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.215246 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97576f7-0dd7-4515-9774-8e8849cc2c5e-kube-api-access-xb5cr" pod "f97576f7-0dd7-4515-9774-8e8849cc2c5e" (UID: "f97576f7-0dd7-4515-9774-8e8849cc2c5e"). InnerVolumeSpecName "kube-api-access-xb5cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:35.222559 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.222536 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-envoy" pod "2bf37ad5-2491-488a-9cea-5f0a3872f5fb" (UID: "2bf37ad5-2491-488a-9cea-5f0a3872f5fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:35.222559 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.222551 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-podinfo" pod "2bf37ad5-2491-488a-9cea-5f0a3872f5fb" (UID: "2bf37ad5-2491-488a-9cea-5f0a3872f5fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:50:35.222741 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.222560 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-token" pod "2bf37ad5-2491-488a-9cea-5f0a3872f5fb" (UID: "2bf37ad5-2491-488a-9cea-5f0a3872f5fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:35.223888 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.223865 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-podinfo" pod "f97576f7-0dd7-4515-9774-8e8849cc2c5e" (UID: "f97576f7-0dd7-4515-9774-8e8849cc2c5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:50:35.278038 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.277989 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j" podUID="27d88842-3d4c-436a-b613-67e561423024" containerName="payload-pre-processing" containerID="cri-o://e7e3c2454b50d3d5fa62a37e85dc8e04244d160a5bba3bc3ba04363b735585f2" gracePeriod=25 Oct 29 23:50:35.312945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.312866 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:35.312945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.312899 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:35.312945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.312910 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:35.312945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.312919 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:35.312945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.312928 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ghvk\" (UniqueName: \"kubernetes.io/projected/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-kube-api-access-9ghvk\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:35.312945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.312937 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:35.312945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.312946 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb5cr\" (UniqueName: \"kubernetes.io/projected/f97576f7-0dd7-4515-9774-8e8849cc2c5e-kube-api-access-xb5cr\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:35.313307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.312954 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:35.313307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.312962 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:35.313307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.312969 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:35.313307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.312977 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97576f7-0dd7-4515-9774-8e8849cc2c5e-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:35.313307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.312985 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf37ad5-2491-488a-9cea-5f0a3872f5fb-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:35.971470 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.971437 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" Oct 29 23:50:35.971917 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.971438 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw" event={"ID":"2bf37ad5-2491-488a-9cea-5f0a3872f5fb","Type":"ContainerDied","Data":"74416197d46fd1d40abdccfff2a98e5f6d1eefc0a1466abaecfd3dfc0becd8b0"} Oct 29 23:50:35.971917 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.971593 5057 scope.go:122] "RemoveContainer" containerID="48261d81e8d198bc853f0369345494e3611da5dd69f97a79e36e0637ac1fb174" Oct 29 23:50:35.973434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.973407 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" event={"ID":"f97576f7-0dd7-4515-9774-8e8849cc2c5e","Type":"ContainerDied","Data":"0b11b60ea71bcdbdd7fd17afbcaa9cd98b6856c30c8be75797c5b94dc0761e6d"} Oct 29 23:50:35.973580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.973560 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j" Oct 29 23:50:35.983103 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:35.983079 5057 scope.go:122] "RemoveContainer" containerID="e84bfaeb37855ca31279d661e919c38f573366c6cb038f3f0577484fda345a24" Oct 29 23:50:36.027827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:36.027798 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j"] Oct 29 23:50:36.042460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:36.042389 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-7bs5j"] Oct 29 23:50:36.069778 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:36.069743 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw"] Oct 29 23:50:36.073350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:36.073323 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-fcvtw"] Oct 29 23:50:37.485680 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:37.485648 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf37ad5-2491-488a-9cea-5f0a3872f5fb" path="/var/lib/kubelet/pods/2bf37ad5-2491-488a-9cea-5f0a3872f5fb/volumes" Oct 29 23:50:37.486244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:37.486166 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f97576f7-0dd7-4515-9774-8e8849cc2c5e" path="/var/lib/kubelet/pods/f97576f7-0dd7-4515-9774-8e8849cc2c5e/volumes" Oct 29 23:50:39.280570 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:39.280540 5057 scope.go:122] "RemoveContainer" containerID="bd9108463c34b48f8ec48e2128b8955fc67f20ada8e4809004bc00c5a4e5b3f8" Oct 29 23:50:42.212209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:42.208686 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j" podUID="27d88842-3d4c-436a-b613-67e561423024" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:50:45.870708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:45.870666 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4"] Oct 29 23:50:46.134302 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:46.133644 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" podUID="3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" containerName="istio-proxy" containerID="cri-o://98fdff5627f6a62754cdbbc651f18a212148fcc5c5f306081792629854cf72e2" gracePeriod=30 Oct 29 23:50:47.501716 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:47.501681 5057 patch_prober.go:29] interesting pod/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:50:47.502122 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:47.501762 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" podUID="3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:50:51.198444 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:51.197356 5057 generic.go:356] "Generic (PLEG): container finished" podID="3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" containerID="98fdff5627f6a62754cdbbc651f18a212148fcc5c5f306081792629854cf72e2" exitCode=0 Oct 29 23:50:51.198444 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:51.197400 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" event={"ID":"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc","Type":"ContainerDied","Data":"98fdff5627f6a62754cdbbc651f18a212148fcc5c5f306081792629854cf72e2"} Oct 29 23:50:51.728825 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:51.728423 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:50:51.728825 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:51.728506 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" podUID="f99eb02d-30df-4333-b4a3-ea8b5ff576cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:50:51.917334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:51.917300 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:50:52.048271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.048140 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-data\") pod \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " Oct 29 23:50:52.050146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.048601 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-kube-api-access-wh4jc\" (UniqueName: \"kubernetes.io/projected/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-kube-api-access-wh4jc\") pod \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " Oct 29 23:50:52.050146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.048699 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-token\" (UniqueName: \"kubernetes.io/projected/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-token\") pod \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " Oct 29 23:50:52.050146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.048805 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istiod-ca-cert\") pod \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " Oct 29 23:50:52.050146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.048869 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-credential-socket\") pod \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " Oct 29 23:50:52.050146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.048903 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-podinfo\") pod \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " Oct 29 23:50:52.050146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.048948 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-workload-certs\") pod \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " Oct 29 23:50:52.050146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.049012 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-workload-socket\") pod \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " Oct 29 23:50:52.050146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.049050 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-envoy\") pod \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\" (UID: \"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc\") " Oct 29 23:50:52.051440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.051104 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-credential-socket" pod "3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" (UID: "3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:52.051440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.051365 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-workload-socket" pod "3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" (UID: "3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:52.051440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.051394 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-data" pod "3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" (UID: "3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:52.051871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.051844 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-workload-certs" pod "3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" (UID: "3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:52.053756 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.053737 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istiod-ca-cert" pod "3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" (UID: "3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:50:52.054013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.053975 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-envoy" pod "3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" (UID: "3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:52.055531 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.054858 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-kube-api-access-wh4jc" pod "3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" (UID: "3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc"). InnerVolumeSpecName "kube-api-access-wh4jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:52.056726 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.056453 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-podinfo" pod "3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" (UID: "3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:50:52.058600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.058552 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-token" pod "3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" (UID: "3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:52.152169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.152123 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:52.152169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.152165 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh4jc\" (UniqueName: \"kubernetes.io/projected/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-kube-api-access-wh4jc\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:52.152169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.152199 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:52.152526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.152216 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:52.152526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.152230 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:52.152526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.152243 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:52.152526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.152256 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:52.152526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.152269 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:52.152526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.152281 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:52.187208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.187155 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j" podUID="27d88842-3d4c-436a-b613-67e561423024" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:50:52.207089 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.207052 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" Oct 29 23:50:52.207089 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.207071 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4" event={"ID":"3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc","Type":"ContainerDied","Data":"4fedd2f3a1075ca6205b17cfd15ef1a5bda1fe52722d8aa3a808b61f27ced2b8"} Oct 29 23:50:52.207616 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.207145 5057 scope.go:122] "RemoveContainer" containerID="98fdff5627f6a62754cdbbc651f18a212148fcc5c5f306081792629854cf72e2" Oct 29 23:50:52.267654 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.267586 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4"] Oct 29 23:50:52.281669 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.279565 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-b7lv4"] Oct 29 23:50:52.860210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.859400 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2"] Oct 29 23:50:52.860517 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860489 5057 cpu_manager.go:414] "containerMap: removing container" podUID="72f6fe5b-dae2-4624-b4f4-bb329f370b87" containerName="istio-proxy" Oct 29 23:50:52.860517 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860513 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f6fe5b-dae2-4624-b4f4-bb329f370b87" containerName="istio-proxy" Oct 29 23:50:52.860708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860541 5057 cpu_manager.go:414] "containerMap: removing container" podUID="ea5a1a45-f903-4dcb-8d39-fabd9bf226e4" containerName="istio-proxy" Oct 29 23:50:52.860708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860549 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5a1a45-f903-4dcb-8d39-fabd9bf226e4" containerName="istio-proxy" Oct 29 23:50:52.860708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860563 5057 cpu_manager.go:414] "containerMap: removing container" podUID="139df2e5-deb8-44b0-8ed5-fb76650beb06" containerName="payload-pre-processing" Oct 29 23:50:52.860708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860572 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="139df2e5-deb8-44b0-8ed5-fb76650beb06" containerName="payload-pre-processing" Oct 29 23:50:52.860708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860589 5057 cpu_manager.go:414] "containerMap: removing container" podUID="895c9cf8-a426-45b5-b9b8-ea94ad9b407b" containerName="istio-proxy" Oct 29 23:50:52.860708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860597 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895c9cf8-a426-45b5-b9b8-ea94ad9b407b" containerName="istio-proxy" Oct 29 23:50:52.860708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860610 5057 cpu_manager.go:414] "containerMap: removing container" podUID="fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" containerName="istio-proxy" Oct 29 23:50:52.860708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860618 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" containerName="istio-proxy" Oct 29 23:50:52.860708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860634 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f97576f7-0dd7-4515-9774-8e8849cc2c5e" containerName="istio-proxy" Oct 29 23:50:52.860708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860642 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97576f7-0dd7-4515-9774-8e8849cc2c5e" containerName="istio-proxy" Oct 29 23:50:52.860708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860659 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1eaa234f-64b4-4698-bf92-c0c03de0e701" containerName="payload-pre-processing" Oct 29 23:50:52.860708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860666 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eaa234f-64b4-4698-bf92-c0c03de0e701" containerName="payload-pre-processing" Oct 29 23:50:52.860708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860679 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d0787860-0815-4f18-9424-cd3ab9944810" containerName="payload-processing" Oct 29 23:50:52.860708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860687 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0787860-0815-4f18-9424-cd3ab9944810" containerName="payload-processing" Oct 29 23:50:52.860708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860700 5057 cpu_manager.go:414] "containerMap: removing container" podUID="fd6e00d3-c546-4fec-82fd-98c420fcca2d" containerName="istio-proxy" Oct 29 23:50:52.860708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860708 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd6e00d3-c546-4fec-82fd-98c420fcca2d" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860725 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1ec6c14d-ff01-457a-abaf-8eab25418539" containerName="payload-processing" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860732 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec6c14d-ff01-457a-abaf-8eab25418539" containerName="payload-processing" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860747 5057 cpu_manager.go:414] "containerMap: removing container" podUID="10e4f8ea-9a96-4cd7-875e-e9ee710c6515" containerName="maas-api" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860754 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e4f8ea-9a96-4cd7-875e-e9ee710c6515" containerName="maas-api" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860771 5057 cpu_manager.go:414] "containerMap: removing container" podUID="895c9cf8-a426-45b5-b9b8-ea94ad9b407b" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860780 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895c9cf8-a426-45b5-b9b8-ea94ad9b407b" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860789 5057 cpu_manager.go:414] "containerMap: removing container" podUID="fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860797 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860814 5057 cpu_manager.go:414] "containerMap: removing container" podUID="ed5ee942-63e4-448f-b180-6790f887ffa6" containerName="payload-processing" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860821 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed5ee942-63e4-448f-b180-6790f887ffa6" containerName="payload-processing" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860839 5057 cpu_manager.go:414] "containerMap: removing container" podUID="3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860847 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860862 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9c0197f2-eb04-41dc-8a00-a3f97ee557d3" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860869 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c0197f2-eb04-41dc-8a00-a3f97ee557d3" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860882 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860889 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860900 5057 cpu_manager.go:414] "containerMap: removing container" podUID="6225f8f9-8cbb-46cb-9563-311fab3d5e21" containerName="payload-pre-processing" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860907 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6225f8f9-8cbb-46cb-9563-311fab3d5e21" containerName="payload-pre-processing" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860924 5057 cpu_manager.go:414] "containerMap: removing container" podUID="772cd9de-1148-400d-9c3b-f0fb52455b8b" containerName="maas-api" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860932 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772cd9de-1148-400d-9c3b-f0fb52455b8b" containerName="maas-api" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860941 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9c0197f2-eb04-41dc-8a00-a3f97ee557d3" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860950 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c0197f2-eb04-41dc-8a00-a3f97ee557d3" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860964 5057 cpu_manager.go:414] "containerMap: removing container" podUID="2bf37ad5-2491-488a-9cea-5f0a3872f5fb" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860972 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf37ad5-2491-488a-9cea-5f0a3872f5fb" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860985 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5d1794a4-da94-40aa-80a4-9d62d8a809ca" containerName="maas-api" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.860992 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d1794a4-da94-40aa-80a4-9d62d8a809ca" containerName="maas-api" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861005 5057 cpu_manager.go:414] "containerMap: removing container" podUID="88075736-9838-476b-8cdd-36014547ca59" containerName="maas-api" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861012 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88075736-9838-476b-8cdd-36014547ca59" containerName="maas-api" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861049 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d89378ad-ddab-4899-9041-7c4cabafa109" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861057 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89378ad-ddab-4899-9041-7c4cabafa109" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861069 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d89378ad-ddab-4899-9041-7c4cabafa109" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861076 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89378ad-ddab-4899-9041-7c4cabafa109" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861255 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eaa234f-64b4-4698-bf92-c0c03de0e701" containerName="payload-pre-processing" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861271 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0787860-0815-4f18-9424-cd3ab9944810" containerName="payload-processing" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861285 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd6e00d3-c546-4fec-82fd-98c420fcca2d" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861296 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec6c14d-ff01-457a-abaf-8eab25418539" containerName="payload-processing" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861305 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e4f8ea-9a96-4cd7-875e-e9ee710c6515" containerName="maas-api" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861318 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="895c9cf8-a426-45b5-b9b8-ea94ad9b407b" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861326 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861335 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed5ee942-63e4-448f-b180-6790f887ffa6" containerName="payload-processing" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861345 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861357 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c0197f2-eb04-41dc-8a00-a3f97ee557d3" containerName="istio-proxy" Oct 29 23:50:52.861345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861366 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eee59f4-800b-4a65-a111-f11fe3afdafe" containerName="main" Oct 29 23:50:52.862793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861375 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="6225f8f9-8cbb-46cb-9563-311fab3d5e21" containerName="payload-pre-processing" Oct 29 23:50:52.862793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861384 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="772cd9de-1148-400d-9c3b-f0fb52455b8b" containerName="maas-api" Oct 29 23:50:52.862793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861393 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf37ad5-2491-488a-9cea-5f0a3872f5fb" containerName="istio-proxy" Oct 29 23:50:52.862793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861403 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d1794a4-da94-40aa-80a4-9d62d8a809ca" containerName="maas-api" Oct 29 23:50:52.862793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861412 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="88075736-9838-476b-8cdd-36014547ca59" containerName="maas-api" Oct 29 23:50:52.862793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861422 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89378ad-ddab-4899-9041-7c4cabafa109" containerName="istio-proxy" Oct 29 23:50:52.862793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861432 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89378ad-ddab-4899-9041-7c4cabafa109" containerName="istio-proxy" Oct 29 23:50:52.862793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861441 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f6fe5b-dae2-4624-b4f4-bb329f370b87" containerName="istio-proxy" Oct 29 23:50:52.862793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861452 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5a1a45-f903-4dcb-8d39-fabd9bf226e4" containerName="istio-proxy" Oct 29 23:50:52.862793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861462 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="139df2e5-deb8-44b0-8ed5-fb76650beb06" containerName="payload-pre-processing" Oct 29 23:50:52.862793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861502 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="895c9cf8-a426-45b5-b9b8-ea94ad9b407b" containerName="istio-proxy" Oct 29 23:50:52.862793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.861511 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97576f7-0dd7-4515-9774-8e8849cc2c5e" containerName="istio-proxy" Oct 29 23:50:52.867285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.867265 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:52.893554 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.893520 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2"] Oct 29 23:50:52.965784 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.965743 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:52.965962 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.965817 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:52.966012 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.965974 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:52.966077 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.966039 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:52.966163 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.966144 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:52.966271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.966255 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/804a9cae-91df-441a-a59d-e8c117344b78-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:52.966354 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.966340 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/804a9cae-91df-441a-a59d-e8c117344b78-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:52.966497 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.966476 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jklbs\" (UniqueName: \"kubernetes.io/projected/804a9cae-91df-441a-a59d-e8c117344b78-kube-api-access-jklbs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:52.966595 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:52.966552 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/804a9cae-91df-441a-a59d-e8c117344b78-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:53.067660 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.067622 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/804a9cae-91df-441a-a59d-e8c117344b78-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:53.067859 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.067722 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jklbs\" (UniqueName: \"kubernetes.io/projected/804a9cae-91df-441a-a59d-e8c117344b78-kube-api-access-jklbs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:53.067859 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.067792 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/804a9cae-91df-441a-a59d-e8c117344b78-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:53.067859 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.067825 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:53.068050 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.067876 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:53.068050 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.067960 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:53.068050 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.068008 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:53.068258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.068056 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:53.068258 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.068124 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/804a9cae-91df-441a-a59d-e8c117344b78-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:53.068400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.068344 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:53.068626 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.068598 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:53.068758 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.068627 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/804a9cae-91df-441a-a59d-e8c117344b78-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:53.068828 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.068749 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:53.068888 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.068868 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:53.071080 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.071055 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:53.071425 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.071404 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/804a9cae-91df-441a-a59d-e8c117344b78-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:53.091353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.091320 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/804a9cae-91df-441a-a59d-e8c117344b78-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:53.091498 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.091461 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jklbs\" (UniqueName: \"kubernetes.io/projected/804a9cae-91df-441a-a59d-e8c117344b78-kube-api-access-jklbs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:53.186292 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.186261 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:53.489359 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.489280 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc" path="/var/lib/kubelet/pods/3b47a487-a766-4e9b-b3d4-c9c94e4c8cdc/volumes" Oct 29 23:50:53.537458 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:50:53.537417 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804a9cae_91df_441a_a59d_e8c117344b78.slice/crio-6d67f2b5ad03b8e454b7811e77ba69b37c7af91ab437d2b9282e14248493b228 WatchSource:0}: Error finding container 6d67f2b5ad03b8e454b7811e77ba69b37c7af91ab437d2b9282e14248493b228: Status 404 returned error can't find the container with id 6d67f2b5ad03b8e454b7811e77ba69b37c7af91ab437d2b9282e14248493b228 Oct 29 23:50:53.538686 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.538659 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2"] Oct 29 23:50:53.540572 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.540532 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:50:53.540686 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.540609 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:50:53.540686 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:53.540664 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:50:54.228011 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:54.227972 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" event={"ID":"804a9cae-91df-441a-a59d-e8c117344b78","Type":"ContainerStarted","Data":"97af36718ab43b288f901767f79cf6812d6908afdcd2179dde6719f80be869aa"} Oct 29 23:50:54.228223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:54.228037 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" event={"ID":"804a9cae-91df-441a-a59d-e8c117344b78","Type":"ContainerStarted","Data":"6d67f2b5ad03b8e454b7811e77ba69b37c7af91ab437d2b9282e14248493b228"} Oct 29 23:50:54.273331 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:54.273272 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" podStartSLOduration=2.273253324 podStartE2EDuration="2.273253324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:50:54.272875096 +0000 UTC m=+4100.924373626" watchObservedRunningTime="2026-10-29 23:50:54.273253324 +0000 UTC m=+4100.924751854" Oct 29 23:50:55.186815 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:55.186781 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:50:55.188448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:55.188421 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Oct 29 23:50:55.188573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:55.188480 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" podUID="804a9cae-91df-441a-a59d-e8c117344b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Oct 29 23:50:55.256340 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:55.256311 5057 generic.go:356] "Generic (PLEG): container finished" podID="f99eb02d-30df-4333-b4a3-ea8b5ff576cf" containerID="c0745d030ae4907be62e5ae3868c41192fc928a3852e69e880a5b6640cda036a" exitCode=0 Oct 29 23:50:55.256556 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:55.256359 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" event={"ID":"f99eb02d-30df-4333-b4a3-ea8b5ff576cf","Type":"ContainerDied","Data":"c0745d030ae4907be62e5ae3868c41192fc928a3852e69e880a5b6640cda036a"} Oct 29 23:50:55.256916 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:55.256895 5057 scope.go:122] "RemoveContainer" containerID="c0745d030ae4907be62e5ae3868c41192fc928a3852e69e880a5b6640cda036a" Oct 29 23:50:55.259676 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:55.259634 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:50:55.259955 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:55.259931 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:50:55.260149 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:55.260129 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:50:55.764976 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:55.764894 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j" Oct 29 23:50:55.832208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:55.820018 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27d88842-3d4c-436a-b613-67e561423024-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27d88842-3d4c-436a-b613-67e561423024-plugins-config-volume\") pod \"27d88842-3d4c-436a-b613-67e561423024\" (UID: \"27d88842-3d4c-436a-b613-67e561423024\") " Oct 29 23:50:55.832208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:55.820552 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d88842-3d4c-436a-b613-67e561423024-kube-api-access-cf2zb\" (UniqueName: \"kubernetes.io/projected/27d88842-3d4c-436a-b613-67e561423024-kube-api-access-cf2zb\") pod \"27d88842-3d4c-436a-b613-67e561423024\" (UID: \"27d88842-3d4c-436a-b613-67e561423024\") " Oct 29 23:50:55.834816 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:55.834787 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d88842-3d4c-436a-b613-67e561423024-kube-api-access-cf2zb" pod "27d88842-3d4c-436a-b613-67e561423024" (UID: "27d88842-3d4c-436a-b613-67e561423024"). InnerVolumeSpecName "kube-api-access-cf2zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:55.857210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:55.836116 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d88842-3d4c-436a-b613-67e561423024-plugins-config-volume" pod "27d88842-3d4c-436a-b613-67e561423024" (UID: "27d88842-3d4c-436a-b613-67e561423024"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:50:55.944211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:55.923528 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27d88842-3d4c-436a-b613-67e561423024-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:55.944211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:55.923566 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf2zb\" (UniqueName: \"kubernetes.io/projected/27d88842-3d4c-436a-b613-67e561423024-kube-api-access-cf2zb\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:50:56.186940 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:56.186906 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Oct 29 23:50:56.187348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:56.186957 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" podUID="804a9cae-91df-441a-a59d-e8c117344b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Oct 29 23:50:56.266459 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:56.266419 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" event={"ID":"f99eb02d-30df-4333-b4a3-ea8b5ff576cf","Type":"ContainerStarted","Data":"cdf8afb7ba2d417f0d3a9437c006569a44b9bca8fc64fcb78b84d9f2c0506557"} Oct 29 23:50:56.268072 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:56.268042 5057 generic.go:356] "Generic (PLEG): container finished" podID="27d88842-3d4c-436a-b613-67e561423024" containerID="e7e3c2454b50d3d5fa62a37e85dc8e04244d160a5bba3bc3ba04363b735585f2" exitCode=0 Oct 29 23:50:56.268230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:56.268095 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j" Oct 29 23:50:56.268230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:56.268123 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j" event={"ID":"27d88842-3d4c-436a-b613-67e561423024","Type":"ContainerDied","Data":"e7e3c2454b50d3d5fa62a37e85dc8e04244d160a5bba3bc3ba04363b735585f2"} Oct 29 23:50:56.268230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:56.268215 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j" event={"ID":"27d88842-3d4c-436a-b613-67e561423024","Type":"ContainerDied","Data":"cd3dbb91b91580588de878c8abf69c65bb19d89f0712bd230f90f372d07b2645"} Oct 29 23:50:56.268363 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:56.268255 5057 scope.go:122] "RemoveContainer" containerID="e7e3c2454b50d3d5fa62a37e85dc8e04244d160a5bba3bc3ba04363b735585f2" Oct 29 23:50:56.284023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:56.284006 5057 scope.go:122] "RemoveContainer" containerID="e7e3c2454b50d3d5fa62a37e85dc8e04244d160a5bba3bc3ba04363b735585f2" Oct 29 23:50:56.284326 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:50:56.284303 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e3c2454b50d3d5fa62a37e85dc8e04244d160a5bba3bc3ba04363b735585f2\": container with ID starting with e7e3c2454b50d3d5fa62a37e85dc8e04244d160a5bba3bc3ba04363b735585f2 not found: ID does not exist" containerID="e7e3c2454b50d3d5fa62a37e85dc8e04244d160a5bba3bc3ba04363b735585f2" Oct 29 23:50:56.284400 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:56.284337 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e3c2454b50d3d5fa62a37e85dc8e04244d160a5bba3bc3ba04363b735585f2"} err="failed to get container status \"e7e3c2454b50d3d5fa62a37e85dc8e04244d160a5bba3bc3ba04363b735585f2\": rpc error: code = NotFound desc = could not find container \"e7e3c2454b50d3d5fa62a37e85dc8e04244d160a5bba3bc3ba04363b735585f2\": container with ID starting with e7e3c2454b50d3d5fa62a37e85dc8e04244d160a5bba3bc3ba04363b735585f2 not found: ID does not exist" Oct 29 23:50:56.361870 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:56.361794 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j"] Oct 29 23:50:56.366412 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:56.366385 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-750972-5bdcf8d96c-vqj5j"] Oct 29 23:50:56.783627 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:56.782646 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:50:56.783949 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:56.783924 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 29 23:50:56.784112 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:56.784087 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" podUID="f99eb02d-30df-4333-b4a3-ea8b5ff576cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 29 23:50:56.901697 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:56.901661 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7"] Oct 29 23:50:57.267226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:57.220413 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Oct 29 23:50:57.267226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:57.220479 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" podUID="804a9cae-91df-441a-a59d-e8c117344b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Oct 29 23:50:57.562262 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:57.558684 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27d88842-3d4c-436a-b613-67e561423024" path="/var/lib/kubelet/pods/27d88842-3d4c-436a-b613-67e561423024/volumes" Oct 29 23:50:57.787210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:57.786642 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 29 23:50:57.787210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:57.786717 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" podUID="f99eb02d-30df-4333-b4a3-ea8b5ff576cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 29 23:50:58.187051 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:58.187015 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Oct 29 23:50:58.187364 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:58.187332 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" podUID="804a9cae-91df-441a-a59d-e8c117344b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Oct 29 23:50:58.789623 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:58.789584 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 29 23:50:58.790282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:58.790248 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" podUID="f99eb02d-30df-4333-b4a3-ea8b5ff576cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 29 23:50:59.187209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:59.186687 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Oct 29 23:50:59.187209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:59.186765 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" podUID="804a9cae-91df-441a-a59d-e8c117344b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Oct 29 23:50:59.782333 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:59.781602 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 29 23:50:59.782333 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:50:59.781687 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" podUID="f99eb02d-30df-4333-b4a3-ea8b5ff576cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 29 23:51:00.186899 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:00.186855 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Oct 29 23:51:00.187332 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:00.186924 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" podUID="804a9cae-91df-441a-a59d-e8c117344b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Oct 29 23:51:00.779994 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:00.779959 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 29 23:51:00.780199 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:00.780013 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" podUID="f99eb02d-30df-4333-b4a3-ea8b5ff576cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 29 23:51:01.191980 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:01.191951 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:51:01.192453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:01.192425 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:51:01.193048 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:01.193027 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:51:01.779964 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:01.779925 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 29 23:51:01.780155 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:01.779995 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" podUID="f99eb02d-30df-4333-b4a3-ea8b5ff576cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 29 23:51:01.902844 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:01.902803 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7" podUID="ed74f944-49b5-453e-8ccb-a46b846e6109" containerName="payload-pre-processing" containerID="cri-o://869d4a164bc9a8532f5c45f9024248fbfd5329fcae05fa2a6721e57c88cb2bf8" gracePeriod=25 Oct 29 23:51:02.184773 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:02.184735 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:51:02.784989 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:02.784950 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:51:02.785482 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:02.785461 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:51:02.786319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:02.786299 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:51:03.917634 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:51:03.917592 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:51:11.185942 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:11.185901 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7" podUID="ed74f944-49b5-453e-8ccb-a46b846e6109" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:51:15.435654 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:15.435617 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw"] Oct 29 23:51:15.466267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:15.466235 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b"] Oct 29 23:51:20.436809 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:20.436765 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw" podUID="ed4c81ea-a9b7-43c6-a893-342cd3d4389f" containerName="payload-processing" containerID="cri-o://c7ed05d5970021f0ea51b7d5fd9456213ffe5c8f73d5e1eafe864750c8607ce2" gracePeriod=25 Oct 29 23:51:20.466568 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:20.466521 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b" podUID="a8aa349b-e04a-4726-9b2e-a910d328e7f9" containerName="payload-pre-processing" containerID="cri-o://baa540b9867d14046559e73916cb37c00e597b42420462eea5f7a2011f7a72a7" gracePeriod=25 Oct 29 23:51:21.186111 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:21.186070 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7" podUID="ed74f944-49b5-453e-8ccb-a46b846e6109" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:51:22.191993 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.191965 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7" Oct 29 23:51:22.346546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.346514 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed74f944-49b5-453e-8ccb-a46b846e6109-kube-api-access-nq7j9\" (UniqueName: \"kubernetes.io/projected/ed74f944-49b5-453e-8ccb-a46b846e6109-kube-api-access-nq7j9\") pod \"ed74f944-49b5-453e-8ccb-a46b846e6109\" (UID: \"ed74f944-49b5-453e-8ccb-a46b846e6109\") " Oct 29 23:51:22.346758 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.346648 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed74f944-49b5-453e-8ccb-a46b846e6109-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed74f944-49b5-453e-8ccb-a46b846e6109-plugins-config-volume\") pod \"ed74f944-49b5-453e-8ccb-a46b846e6109\" (UID: \"ed74f944-49b5-453e-8ccb-a46b846e6109\") " Oct 29 23:51:22.347055 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.347028 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed74f944-49b5-453e-8ccb-a46b846e6109-plugins-config-volume" pod "ed74f944-49b5-453e-8ccb-a46b846e6109" (UID: "ed74f944-49b5-453e-8ccb-a46b846e6109"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:51:22.347355 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.347338 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed74f944-49b5-453e-8ccb-a46b846e6109-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:22.348780 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.348749 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed74f944-49b5-453e-8ccb-a46b846e6109-kube-api-access-nq7j9" pod "ed74f944-49b5-453e-8ccb-a46b846e6109" (UID: "ed74f944-49b5-453e-8ccb-a46b846e6109"). InnerVolumeSpecName "kube-api-access-nq7j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:22.449203 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.449162 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq7j9\" (UniqueName: \"kubernetes.io/projected/ed74f944-49b5-453e-8ccb-a46b846e6109-kube-api-access-nq7j9\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:22.670381 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.670329 5057 generic.go:356] "Generic (PLEG): container finished" podID="ed74f944-49b5-453e-8ccb-a46b846e6109" containerID="869d4a164bc9a8532f5c45f9024248fbfd5329fcae05fa2a6721e57c88cb2bf8" exitCode=0 Oct 29 23:51:22.670690 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.670658 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7" event={"ID":"ed74f944-49b5-453e-8ccb-a46b846e6109","Type":"ContainerDied","Data":"869d4a164bc9a8532f5c45f9024248fbfd5329fcae05fa2a6721e57c88cb2bf8"} Oct 29 23:51:22.670775 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.670738 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7" event={"ID":"ed74f944-49b5-453e-8ccb-a46b846e6109","Type":"ContainerDied","Data":"9cd4b8325306b307595bfabb251ce20ed92208da5f4c00658144136051e4365a"} Oct 29 23:51:22.670775 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.670771 5057 scope.go:122] "RemoveContainer" containerID="869d4a164bc9a8532f5c45f9024248fbfd5329fcae05fa2a6721e57c88cb2bf8" Oct 29 23:51:22.670984 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.670965 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7" Oct 29 23:51:22.692223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.692198 5057 scope.go:122] "RemoveContainer" containerID="869d4a164bc9a8532f5c45f9024248fbfd5329fcae05fa2a6721e57c88cb2bf8" Oct 29 23:51:22.692597 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:51:22.692570 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"869d4a164bc9a8532f5c45f9024248fbfd5329fcae05fa2a6721e57c88cb2bf8\": container with ID starting with 869d4a164bc9a8532f5c45f9024248fbfd5329fcae05fa2a6721e57c88cb2bf8 not found: ID does not exist" containerID="869d4a164bc9a8532f5c45f9024248fbfd5329fcae05fa2a6721e57c88cb2bf8" Oct 29 23:51:22.692699 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.692605 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"869d4a164bc9a8532f5c45f9024248fbfd5329fcae05fa2a6721e57c88cb2bf8"} err="failed to get container status \"869d4a164bc9a8532f5c45f9024248fbfd5329fcae05fa2a6721e57c88cb2bf8\": rpc error: code = NotFound desc = could not find container \"869d4a164bc9a8532f5c45f9024248fbfd5329fcae05fa2a6721e57c88cb2bf8\": container with ID starting with 869d4a164bc9a8532f5c45f9024248fbfd5329fcae05fa2a6721e57c88cb2bf8 not found: ID does not exist" Oct 29 23:51:22.726859 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.726782 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7"] Oct 29 23:51:22.738725 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.738696 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d20345-5876db4fb7-l2ps7"] Oct 29 23:51:22.975016 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.974983 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg"] Oct 29 23:51:22.975809 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.975791 5057 cpu_manager.go:414] "containerMap: removing container" podUID="ed74f944-49b5-453e-8ccb-a46b846e6109" containerName="payload-pre-processing" Oct 29 23:51:22.975809 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.975809 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed74f944-49b5-453e-8ccb-a46b846e6109" containerName="payload-pre-processing" Oct 29 23:51:22.975956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.975835 5057 cpu_manager.go:414] "containerMap: removing container" podUID="fd6e00d3-c546-4fec-82fd-98c420fcca2d" containerName="istio-proxy" Oct 29 23:51:22.975956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.975841 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd6e00d3-c546-4fec-82fd-98c420fcca2d" containerName="istio-proxy" Oct 29 23:51:22.975956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.975850 5057 cpu_manager.go:414] "containerMap: removing container" podUID="27d88842-3d4c-436a-b613-67e561423024" containerName="payload-pre-processing" Oct 29 23:51:22.975956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.975860 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d88842-3d4c-436a-b613-67e561423024" containerName="payload-pre-processing" Oct 29 23:51:22.976106 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.975998 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed74f944-49b5-453e-8ccb-a46b846e6109" containerName="payload-pre-processing" Oct 29 23:51:22.976106 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.976011 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd6e00d3-c546-4fec-82fd-98c420fcca2d" containerName="istio-proxy" Oct 29 23:51:22.976106 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.976022 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d88842-3d4c-436a-b613-67e561423024" containerName="payload-pre-processing" Oct 29 23:51:22.976106 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.976032 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="fefd6fcf-b63a-4d1f-8b0a-8a3e6a033e31" containerName="istio-proxy" Oct 29 23:51:22.976106 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.976039 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c0197f2-eb04-41dc-8a00-a3f97ee557d3" containerName="istio-proxy" Oct 29 23:51:22.981262 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:22.981182 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.007010 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.006967 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg"] Oct 29 23:51:23.065448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.065405 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.065680 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.065459 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.065680 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.065547 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.065680 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.065621 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzlft\" (UniqueName: \"kubernetes.io/projected/1ae60c2b-2e14-416e-b552-fb9b929be559-kube-api-access-fzlft\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.065880 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.065690 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.065880 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.065764 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.065880 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.065849 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.066048 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.065919 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.066125 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.066109 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ae60c2b-2e14-416e-b552-fb9b929be559-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.167798 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.167766 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ae60c2b-2e14-416e-b552-fb9b929be559-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.167970 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.167873 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.167970 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.167898 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.167970 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.167916 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.168120 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.168049 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzlft\" (UniqueName: \"kubernetes.io/projected/1ae60c2b-2e14-416e-b552-fb9b929be559-kube-api-access-fzlft\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.168225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.168120 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.168225 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.168211 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.168348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.168261 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.168348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.168323 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.168488 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.168354 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.168559 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.168539 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.168634 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.168612 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.168678 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.168633 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ae60c2b-2e14-416e-b552-fb9b929be559-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.168732 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.168676 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.170534 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.170503 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.170673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.170634 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.190752 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.190724 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.190921 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.190899 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzlft\" (UniqueName: \"kubernetes.io/projected/1ae60c2b-2e14-416e-b552-fb9b929be559-kube-api-access-fzlft\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.300510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.300427 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:23.487288 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.487251 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed74f944-49b5-453e-8ccb-a46b846e6109" path="/var/lib/kubelet/pods/ed74f944-49b5-453e-8ccb-a46b846e6109/volumes" Oct 29 23:51:23.630845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.630815 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg"] Oct 29 23:51:23.631950 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:51:23.631920 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae60c2b_2e14_416e_b552_fb9b929be559.slice/crio-95f555f77c79eb398fca732db6c6c9946c18b07e9d260f3ab05c630d9ec337c6 WatchSource:0}: Error finding container 95f555f77c79eb398fca732db6c6c9946c18b07e9d260f3ab05c630d9ec337c6: Status 404 returned error can't find the container with id 95f555f77c79eb398fca732db6c6c9946c18b07e9d260f3ab05c630d9ec337c6 Oct 29 23:51:23.635342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.635307 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:51:23.635427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.635377 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:51:23.635427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.635407 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:51:23.681987 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:23.681954 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" event={"ID":"1ae60c2b-2e14-416e-b552-fb9b929be559","Type":"ContainerStarted","Data":"95f555f77c79eb398fca732db6c6c9946c18b07e9d260f3ab05c630d9ec337c6"} Oct 29 23:51:24.693068 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:24.693025 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" event={"ID":"1ae60c2b-2e14-416e-b552-fb9b929be559","Type":"ContainerStarted","Data":"6220b53cd031de6f02748f59a69b2a3ce39d8b6862ee3c1a307df2ec49e9bebe"} Oct 29 23:51:24.753223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:24.753140 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" podStartSLOduration=2.75312494 podStartE2EDuration="2.75312494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:51:24.750636756 +0000 UTC m=+4131.402135289" watchObservedRunningTime="2026-10-29 23:51:24.75312494 +0000 UTC m=+4131.404623541" Oct 29 23:51:25.304134 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:25.304089 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:25.304406 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:25.304359 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Oct 29 23:51:25.304499 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:25.304420 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" podUID="1ae60c2b-2e14-416e-b552-fb9b929be559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Oct 29 23:51:26.300803 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:26.300763 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Oct 29 23:51:26.301326 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:26.300822 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" podUID="1ae60c2b-2e14-416e-b552-fb9b929be559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Oct 29 23:51:27.306210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:27.305357 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Oct 29 23:51:27.306210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:27.305436 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" podUID="1ae60c2b-2e14-416e-b552-fb9b929be559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Oct 29 23:51:28.301694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:28.301659 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Oct 29 23:51:28.301896 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:28.301731 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" podUID="1ae60c2b-2e14-416e-b552-fb9b929be559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Oct 29 23:51:29.300828 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:29.300788 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Oct 29 23:51:29.301272 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:29.300857 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" podUID="1ae60c2b-2e14-416e-b552-fb9b929be559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Oct 29 23:51:30.046903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:30.046860 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b" podUID="a8aa349b-e04a-4726-9b2e-a910d328e7f9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:51:30.047106 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:30.046981 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw" podUID="ed4c81ea-a9b7-43c6-a893-342cd3d4389f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:51:30.306641 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:30.306548 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:30.307117 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:30.307094 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:30.308125 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:30.308103 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:33.914324 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:51:33.914283 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:51:34.670768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:34.670722 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:51:35.192018 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:35.191975 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg"] Oct 29 23:51:35.192992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:35.192957 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" podUID="1ae60c2b-2e14-416e-b552-fb9b929be559" containerName="istio-proxy" containerID="cri-o://6220b53cd031de6f02748f59a69b2a3ce39d8b6862ee3c1a307df2ec49e9bebe" gracePeriod=30 Oct 29 23:51:35.205296 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:35.205263 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m"] Oct 29 23:51:35.205545 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:35.205517 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" podUID="f99eb02d-30df-4333-b4a3-ea8b5ff576cf" containerName="istio-proxy" containerID="cri-o://cdf8afb7ba2d417f0d3a9437c006569a44b9bca8fc64fcb78b84d9f2c0506557" gracePeriod=30 Oct 29 23:51:35.211993 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:35.211966 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2"] Oct 29 23:51:35.212259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:35.212230 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" podUID="804a9cae-91df-441a-a59d-e8c117344b78" containerName="istio-proxy" containerID="cri-o://97af36718ab43b288f901767f79cf6812d6908afdcd2179dde6719f80be869aa" gracePeriod=30 Oct 29 23:51:40.082101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.082056 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw" podUID="ed4c81ea-a9b7-43c6-a893-342cd3d4389f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:51:40.083124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.082275 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b" podUID="a8aa349b-e04a-4726-9b2e-a910d328e7f9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:51:40.609609 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.609244 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:51:40.693031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.690765 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-envoy\") pod \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " Oct 29 23:51:40.693031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.690860 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-data\") pod \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " Oct 29 23:51:40.693031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.690926 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-kube-api-access-g7cfw\" (UniqueName: \"kubernetes.io/projected/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-kube-api-access-g7cfw\") pod \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " Oct 29 23:51:40.693031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.691001 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istiod-ca-cert\") pod \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " Oct 29 23:51:40.693031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.691084 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-credential-socket\") pod \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " Oct 29 23:51:40.693031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.691173 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-workload-socket\") pod \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " Oct 29 23:51:40.693031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.691253 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-workload-certs\") pod \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " Oct 29 23:51:40.693031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.691395 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-podinfo\") pod \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " Oct 29 23:51:40.693031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.691494 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-token\" (UniqueName: \"kubernetes.io/projected/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-token\") pod \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\" (UID: \"f99eb02d-30df-4333-b4a3-ea8b5ff576cf\") " Oct 29 23:51:40.757209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.744971 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istiod-ca-cert" pod "f99eb02d-30df-4333-b4a3-ea8b5ff576cf" (UID: "f99eb02d-30df-4333-b4a3-ea8b5ff576cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:51:40.757209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.747584 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-data" pod "f99eb02d-30df-4333-b4a3-ea8b5ff576cf" (UID: "f99eb02d-30df-4333-b4a3-ea8b5ff576cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:40.757209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.748334 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:40.757209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.748955 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-credential-socket" pod "f99eb02d-30df-4333-b4a3-ea8b5ff576cf" (UID: "f99eb02d-30df-4333-b4a3-ea8b5ff576cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:40.757209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.750634 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.757209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.750659 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.757209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.750673 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.757209 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.751266 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-workload-certs" pod "f99eb02d-30df-4333-b4a3-ea8b5ff576cf" (UID: "f99eb02d-30df-4333-b4a3-ea8b5ff576cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:40.759325 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.759269 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-workload-socket" pod "f99eb02d-30df-4333-b4a3-ea8b5ff576cf" (UID: "f99eb02d-30df-4333-b4a3-ea8b5ff576cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:40.790035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.789117 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc"] Oct 29 23:51:40.841313 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.840785 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-token" pod "f99eb02d-30df-4333-b4a3-ea8b5ff576cf" (UID: "f99eb02d-30df-4333-b4a3-ea8b5ff576cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:40.857267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.852443 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-data\") pod \"1ae60c2b-2e14-416e-b552-fb9b929be559\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " Oct 29 23:51:40.871250 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.871222 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-credential-socket\") pod \"1ae60c2b-2e14-416e-b552-fb9b929be559\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " Oct 29 23:51:40.872019 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.871985 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae60c2b-2e14-416e-b552-fb9b929be559-kube-api-access-fzlft\" (UniqueName: \"kubernetes.io/projected/1ae60c2b-2e14-416e-b552-fb9b929be559-kube-api-access-fzlft\") pod \"1ae60c2b-2e14-416e-b552-fb9b929be559\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " Oct 29 23:51:40.872558 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.871907 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-credential-socket" pod "1ae60c2b-2e14-416e-b552-fb9b929be559" (UID: "1ae60c2b-2e14-416e-b552-fb9b929be559"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:40.873171 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.872750 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-token\" (UniqueName: \"kubernetes.io/projected/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-token\") pod \"1ae60c2b-2e14-416e-b552-fb9b929be559\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " Oct 29 23:51:40.873682 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.873635 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae60c2b-2e14-416e-b552-fb9b929be559-istiod-ca-cert" pod "1ae60c2b-2e14-416e-b552-fb9b929be559" (UID: "1ae60c2b-2e14-416e-b552-fb9b929be559"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:51:40.873870 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.873813 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae60c2b-2e14-416e-b552-fb9b929be559-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ae60c2b-2e14-416e-b552-fb9b929be559-istiod-ca-cert\") pod \"1ae60c2b-2e14-416e-b552-fb9b929be559\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " Oct 29 23:51:40.874389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.874021 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-workload-socket\") pod \"1ae60c2b-2e14-416e-b552-fb9b929be559\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " Oct 29 23:51:40.874389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.874335 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-workload-socket" pod "1ae60c2b-2e14-416e-b552-fb9b929be559" (UID: "1ae60c2b-2e14-416e-b552-fb9b929be559"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:40.874969 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.874715 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-workload-certs\") pod \"1ae60c2b-2e14-416e-b552-fb9b929be559\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " Oct 29 23:51:40.875113 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.874930 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-workload-certs" pod "1ae60c2b-2e14-416e-b552-fb9b929be559" (UID: "1ae60c2b-2e14-416e-b552-fb9b929be559"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:40.875738 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.875413 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-podinfo\") pod \"1ae60c2b-2e14-416e-b552-fb9b929be559\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " Oct 29 23:51:40.879720 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.875928 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-envoy\") pod \"1ae60c2b-2e14-416e-b552-fb9b929be559\" (UID: \"1ae60c2b-2e14-416e-b552-fb9b929be559\") " Oct 29 23:51:40.883339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.883227 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.883339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.883249 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.883339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.883262 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.883339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.883274 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ae60c2b-2e14-416e-b552-fb9b929be559-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.883339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.883290 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.883339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.883300 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.883339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.883312 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.887819 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.887737 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-data" pod "1ae60c2b-2e14-416e-b552-fb9b929be559" (UID: "1ae60c2b-2e14-416e-b552-fb9b929be559"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:40.917521 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.915585 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-kube-api-access-g7cfw" pod "f99eb02d-30df-4333-b4a3-ea8b5ff576cf" (UID: "f99eb02d-30df-4333-b4a3-ea8b5ff576cf"). InnerVolumeSpecName "kube-api-access-g7cfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:40.927567 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.922316 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-envoy" pod "f99eb02d-30df-4333-b4a3-ea8b5ff576cf" (UID: "f99eb02d-30df-4333-b4a3-ea8b5ff576cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:40.936249 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.931562 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-podinfo" pod "f99eb02d-30df-4333-b4a3-ea8b5ff576cf" (UID: "f99eb02d-30df-4333-b4a3-ea8b5ff576cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:51:40.951542 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.951422 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-podinfo" pod "1ae60c2b-2e14-416e-b552-fb9b929be559" (UID: "1ae60c2b-2e14-416e-b552-fb9b929be559"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:51:40.951542 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.951509 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-envoy" pod "1ae60c2b-2e14-416e-b552-fb9b929be559" (UID: "1ae60c2b-2e14-416e-b552-fb9b929be559"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:40.955805 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.955769 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae60c2b-2e14-416e-b552-fb9b929be559-kube-api-access-fzlft" pod "1ae60c2b-2e14-416e-b552-fb9b929be559" (UID: "1ae60c2b-2e14-416e-b552-fb9b929be559"). InnerVolumeSpecName "kube-api-access-fzlft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:40.965530 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.965495 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-token" pod "1ae60c2b-2e14-416e-b552-fb9b929be559" (UID: "1ae60c2b-2e14-416e-b552-fb9b929be559"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:40.968793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.968410 5057 generic.go:356] "Generic (PLEG): container finished" podID="a8aa349b-e04a-4726-9b2e-a910d328e7f9" containerID="baa540b9867d14046559e73916cb37c00e597b42420462eea5f7a2011f7a72a7" exitCode=0 Oct 29 23:51:40.968793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.968579 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b" event={"ID":"a8aa349b-e04a-4726-9b2e-a910d328e7f9","Type":"ContainerDied","Data":"baa540b9867d14046559e73916cb37c00e597b42420462eea5f7a2011f7a72a7"} Oct 29 23:51:40.971802 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.970894 5057 generic.go:356] "Generic (PLEG): container finished" podID="ed4c81ea-a9b7-43c6-a893-342cd3d4389f" containerID="c7ed05d5970021f0ea51b7d5fd9456213ffe5c8f73d5e1eafe864750c8607ce2" exitCode=0 Oct 29 23:51:40.971802 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.970999 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw" event={"ID":"ed4c81ea-a9b7-43c6-a893-342cd3d4389f","Type":"ContainerDied","Data":"c7ed05d5970021f0ea51b7d5fd9456213ffe5c8f73d5e1eafe864750c8607ce2"} Oct 29 23:51:40.976212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.975272 5057 generic.go:356] "Generic (PLEG): container finished" podID="1ae60c2b-2e14-416e-b552-fb9b929be559" containerID="6220b53cd031de6f02748f59a69b2a3ce39d8b6862ee3c1a307df2ec49e9bebe" exitCode=0 Oct 29 23:51:40.976212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.975452 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" Oct 29 23:51:40.980598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.980267 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" event={"ID":"1ae60c2b-2e14-416e-b552-fb9b929be559","Type":"ContainerDied","Data":"6220b53cd031de6f02748f59a69b2a3ce39d8b6862ee3c1a307df2ec49e9bebe"} Oct 29 23:51:40.980598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.980319 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg" event={"ID":"1ae60c2b-2e14-416e-b552-fb9b929be559","Type":"ContainerDied","Data":"95f555f77c79eb398fca732db6c6c9946c18b07e9d260f3ab05c630d9ec337c6"} Oct 29 23:51:40.980598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.980351 5057 scope.go:122] "RemoveContainer" containerID="6220b53cd031de6f02748f59a69b2a3ce39d8b6862ee3c1a307df2ec49e9bebe" Oct 29 23:51:40.986414 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.985398 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.986414 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.985427 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.986414 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.985440 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7cfw\" (UniqueName: \"kubernetes.io/projected/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-kube-api-access-g7cfw\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.986414 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.985452 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.986414 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.985466 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.986414 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.985480 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzlft\" (UniqueName: \"kubernetes.io/projected/1ae60c2b-2e14-416e-b552-fb9b929be559-kube-api-access-fzlft\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.986414 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.985491 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ae60c2b-2e14-416e-b552-fb9b929be559-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.986414 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.985503 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f99eb02d-30df-4333-b4a3-ea8b5ff576cf-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.991068 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.990692 5057 generic.go:356] "Generic (PLEG): container finished" podID="804a9cae-91df-441a-a59d-e8c117344b78" containerID="97af36718ab43b288f901767f79cf6812d6908afdcd2179dde6719f80be869aa" exitCode=0 Oct 29 23:51:40.991068 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.990780 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" event={"ID":"804a9cae-91df-441a-a59d-e8c117344b78","Type":"ContainerDied","Data":"97af36718ab43b288f901767f79cf6812d6908afdcd2179dde6719f80be869aa"} Oct 29 23:51:40.994370 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.993208 5057 generic.go:356] "Generic (PLEG): container finished" podID="f99eb02d-30df-4333-b4a3-ea8b5ff576cf" containerID="cdf8afb7ba2d417f0d3a9437c006569a44b9bca8fc64fcb78b84d9f2c0506557" exitCode=0 Oct 29 23:51:40.994370 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.993259 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" event={"ID":"f99eb02d-30df-4333-b4a3-ea8b5ff576cf","Type":"ContainerDied","Data":"cdf8afb7ba2d417f0d3a9437c006569a44b9bca8fc64fcb78b84d9f2c0506557"} Oct 29 23:51:40.994370 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.993297 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" event={"ID":"f99eb02d-30df-4333-b4a3-ea8b5ff576cf","Type":"ContainerDied","Data":"9aba5d888f25fb191f6e404e7694e6943c9314f0b6d3a3e47d594c776fcbecac"} Oct 29 23:51:40.994370 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:40.993377 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m" Oct 29 23:51:41.006633 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.006098 5057 scope.go:122] "RemoveContainer" containerID="6220b53cd031de6f02748f59a69b2a3ce39d8b6862ee3c1a307df2ec49e9bebe" Oct 29 23:51:41.006633 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:51:41.006394 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6220b53cd031de6f02748f59a69b2a3ce39d8b6862ee3c1a307df2ec49e9bebe\": container with ID starting with 6220b53cd031de6f02748f59a69b2a3ce39d8b6862ee3c1a307df2ec49e9bebe not found: ID does not exist" containerID="6220b53cd031de6f02748f59a69b2a3ce39d8b6862ee3c1a307df2ec49e9bebe" Oct 29 23:51:41.006633 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.006428 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6220b53cd031de6f02748f59a69b2a3ce39d8b6862ee3c1a307df2ec49e9bebe"} err="failed to get container status \"6220b53cd031de6f02748f59a69b2a3ce39d8b6862ee3c1a307df2ec49e9bebe\": rpc error: code = NotFound desc = could not find container \"6220b53cd031de6f02748f59a69b2a3ce39d8b6862ee3c1a307df2ec49e9bebe\": container with ID starting with 6220b53cd031de6f02748f59a69b2a3ce39d8b6862ee3c1a307df2ec49e9bebe not found: ID does not exist" Oct 29 23:51:41.006633 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.006454 5057 scope.go:122] "RemoveContainer" containerID="cdf8afb7ba2d417f0d3a9437c006569a44b9bca8fc64fcb78b84d9f2c0506557" Oct 29 23:51:41.045620 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.045595 5057 scope.go:122] "RemoveContainer" containerID="c0745d030ae4907be62e5ae3868c41192fc928a3852e69e880a5b6640cda036a" Oct 29 23:51:41.071349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.071269 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg"] Oct 29 23:51:41.075329 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.075246 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:51:41.077883 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.077500 5057 scope.go:122] "RemoveContainer" containerID="cdf8afb7ba2d417f0d3a9437c006569a44b9bca8fc64fcb78b84d9f2c0506557" Oct 29 23:51:41.077883 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:51:41.077859 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf8afb7ba2d417f0d3a9437c006569a44b9bca8fc64fcb78b84d9f2c0506557\": container with ID starting with cdf8afb7ba2d417f0d3a9437c006569a44b9bca8fc64fcb78b84d9f2c0506557 not found: ID does not exist" containerID="cdf8afb7ba2d417f0d3a9437c006569a44b9bca8fc64fcb78b84d9f2c0506557" Oct 29 23:51:41.078011 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.077891 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf8afb7ba2d417f0d3a9437c006569a44b9bca8fc64fcb78b84d9f2c0506557"} err="failed to get container status \"cdf8afb7ba2d417f0d3a9437c006569a44b9bca8fc64fcb78b84d9f2c0506557\": rpc error: code = NotFound desc = could not find container \"cdf8afb7ba2d417f0d3a9437c006569a44b9bca8fc64fcb78b84d9f2c0506557\": container with ID starting with cdf8afb7ba2d417f0d3a9437c006569a44b9bca8fc64fcb78b84d9f2c0506557 not found: ID does not exist" Oct 29 23:51:41.078011 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.077914 5057 scope.go:122] "RemoveContainer" containerID="c0745d030ae4907be62e5ae3868c41192fc928a3852e69e880a5b6640cda036a" Oct 29 23:51:41.078262 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:51:41.078200 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0745d030ae4907be62e5ae3868c41192fc928a3852e69e880a5b6640cda036a\": container with ID starting with c0745d030ae4907be62e5ae3868c41192fc928a3852e69e880a5b6640cda036a not found: ID does not exist" containerID="c0745d030ae4907be62e5ae3868c41192fc928a3852e69e880a5b6640cda036a" Oct 29 23:51:41.078262 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.078230 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0745d030ae4907be62e5ae3868c41192fc928a3852e69e880a5b6640cda036a"} err="failed to get container status \"c0745d030ae4907be62e5ae3868c41192fc928a3852e69e880a5b6640cda036a\": rpc error: code = NotFound desc = could not find container \"c0745d030ae4907be62e5ae3868c41192fc928a3852e69e880a5b6640cda036a\": container with ID starting with c0745d030ae4907be62e5ae3868c41192fc928a3852e69e880a5b6640cda036a not found: ID does not exist" Oct 29 23:51:41.078586 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.078559 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-jzbmg"] Oct 29 23:51:41.092987 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.092940 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b" Oct 29 23:51:41.095663 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.095644 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw" Oct 29 23:51:41.106615 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.106562 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m"] Oct 29 23:51:41.109972 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.109949 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-wkb4m"] Oct 29 23:51:41.187703 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.187671 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed4c81ea-a9b7-43c6-a893-342cd3d4389f-kube-api-access-65brg\" (UniqueName: \"kubernetes.io/projected/ed4c81ea-a9b7-43c6-a893-342cd3d4389f-kube-api-access-65brg\") pod \"ed4c81ea-a9b7-43c6-a893-342cd3d4389f\" (UID: \"ed4c81ea-a9b7-43c6-a893-342cd3d4389f\") " Oct 29 23:51:41.187913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.187713 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-workload-socket\") pod \"804a9cae-91df-441a-a59d-e8c117344b78\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " Oct 29 23:51:41.187913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.187751 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/804a9cae-91df-441a-a59d-e8c117344b78-kube-api-access-jklbs\" (UniqueName: \"kubernetes.io/projected/804a9cae-91df-441a-a59d-e8c117344b78-kube-api-access-jklbs\") pod \"804a9cae-91df-441a-a59d-e8c117344b78\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " Oct 29 23:51:41.187913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.187848 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-credential-socket\") pod \"804a9cae-91df-441a-a59d-e8c117344b78\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " Oct 29 23:51:41.188074 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.188036 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-workload-socket" pod "804a9cae-91df-441a-a59d-e8c117344b78" (UID: "804a9cae-91df-441a-a59d-e8c117344b78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:41.188319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.188292 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-credential-socket" pod "804a9cae-91df-441a-a59d-e8c117344b78" (UID: "804a9cae-91df-441a-a59d-e8c117344b78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:41.188442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.188339 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/804a9cae-91df-441a-a59d-e8c117344b78-istio-token\" (UniqueName: \"kubernetes.io/projected/804a9cae-91df-441a-a59d-e8c117344b78-istio-token\") pod \"804a9cae-91df-441a-a59d-e8c117344b78\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " Oct 29 23:51:41.188442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.188395 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8aa349b-e04a-4726-9b2e-a910d328e7f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8aa349b-e04a-4726-9b2e-a910d328e7f9-plugins-config-volume\") pod \"a8aa349b-e04a-4726-9b2e-a910d328e7f9\" (UID: \"a8aa349b-e04a-4726-9b2e-a910d328e7f9\") " Oct 29 23:51:41.188584 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.188445 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed4c81ea-a9b7-43c6-a893-342cd3d4389f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4c81ea-a9b7-43c6-a893-342cd3d4389f-plugins-config-volume\") pod \"ed4c81ea-a9b7-43c6-a893-342cd3d4389f\" (UID: \"ed4c81ea-a9b7-43c6-a893-342cd3d4389f\") " Oct 29 23:51:41.188584 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.188484 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/804a9cae-91df-441a-a59d-e8c117344b78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/804a9cae-91df-441a-a59d-e8c117344b78-istiod-ca-cert\") pod \"804a9cae-91df-441a-a59d-e8c117344b78\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " Oct 29 23:51:41.188584 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.188550 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-istio-data\") pod \"804a9cae-91df-441a-a59d-e8c117344b78\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " Oct 29 23:51:41.188790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.188647 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-istio-envoy\") pod \"804a9cae-91df-441a-a59d-e8c117344b78\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " Oct 29 23:51:41.188790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.188701 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8aa349b-e04a-4726-9b2e-a910d328e7f9-kube-api-access-4bp4x\" (UniqueName: \"kubernetes.io/projected/a8aa349b-e04a-4726-9b2e-a910d328e7f9-kube-api-access-4bp4x\") pod \"a8aa349b-e04a-4726-9b2e-a910d328e7f9\" (UID: \"a8aa349b-e04a-4726-9b2e-a910d328e7f9\") " Oct 29 23:51:41.188912 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.188791 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-workload-certs\") pod \"804a9cae-91df-441a-a59d-e8c117344b78\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " Oct 29 23:51:41.188912 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.188795 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8aa349b-e04a-4726-9b2e-a910d328e7f9-plugins-config-volume" pod "a8aa349b-e04a-4726-9b2e-a910d328e7f9" (UID: "a8aa349b-e04a-4726-9b2e-a910d328e7f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:51:41.188912 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.188866 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/804a9cae-91df-441a-a59d-e8c117344b78-istiod-ca-cert" pod "804a9cae-91df-441a-a59d-e8c117344b78" (UID: "804a9cae-91df-441a-a59d-e8c117344b78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:51:41.188912 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.188872 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/804a9cae-91df-441a-a59d-e8c117344b78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/804a9cae-91df-441a-a59d-e8c117344b78-istio-podinfo\") pod \"804a9cae-91df-441a-a59d-e8c117344b78\" (UID: \"804a9cae-91df-441a-a59d-e8c117344b78\") " Oct 29 23:51:41.189155 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.188911 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed4c81ea-a9b7-43c6-a893-342cd3d4389f-plugins-config-volume" pod "ed4c81ea-a9b7-43c6-a893-342cd3d4389f" (UID: "ed4c81ea-a9b7-43c6-a893-342cd3d4389f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:51:41.189516 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.189490 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-istio-data" pod "804a9cae-91df-441a-a59d-e8c117344b78" (UID: "804a9cae-91df-441a-a59d-e8c117344b78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:41.189688 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.189668 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-workload-certs" pod "804a9cae-91df-441a-a59d-e8c117344b78" (UID: "804a9cae-91df-441a-a59d-e8c117344b78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:41.189799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.189783 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:41.189875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.189804 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:41.189875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.189819 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8aa349b-e04a-4726-9b2e-a910d328e7f9-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:41.189875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.189833 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4c81ea-a9b7-43c6-a893-342cd3d4389f-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:41.189875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.189845 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/804a9cae-91df-441a-a59d-e8c117344b78-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:41.189875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.189859 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:41.189875 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.189871 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:41.191280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.191247 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-istio-envoy" pod "804a9cae-91df-441a-a59d-e8c117344b78" (UID: "804a9cae-91df-441a-a59d-e8c117344b78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:41.191280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.191259 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed4c81ea-a9b7-43c6-a893-342cd3d4389f-kube-api-access-65brg" pod "ed4c81ea-a9b7-43c6-a893-342cd3d4389f" (UID: "ed4c81ea-a9b7-43c6-a893-342cd3d4389f"). InnerVolumeSpecName "kube-api-access-65brg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:41.191454 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.191406 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/804a9cae-91df-441a-a59d-e8c117344b78-istio-podinfo" pod "804a9cae-91df-441a-a59d-e8c117344b78" (UID: "804a9cae-91df-441a-a59d-e8c117344b78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:51:41.191698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.191675 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804a9cae-91df-441a-a59d-e8c117344b78-kube-api-access-jklbs" pod "804a9cae-91df-441a-a59d-e8c117344b78" (UID: "804a9cae-91df-441a-a59d-e8c117344b78"). InnerVolumeSpecName "kube-api-access-jklbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:41.192089 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.192068 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804a9cae-91df-441a-a59d-e8c117344b78-istio-token" pod "804a9cae-91df-441a-a59d-e8c117344b78" (UID: "804a9cae-91df-441a-a59d-e8c117344b78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:41.192174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.192135 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8aa349b-e04a-4726-9b2e-a910d328e7f9-kube-api-access-4bp4x" pod "a8aa349b-e04a-4726-9b2e-a910d328e7f9" (UID: "a8aa349b-e04a-4726-9b2e-a910d328e7f9"). InnerVolumeSpecName "kube-api-access-4bp4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:41.291651 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.291619 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/804a9cae-91df-441a-a59d-e8c117344b78-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:41.291651 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.291650 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65brg\" (UniqueName: \"kubernetes.io/projected/ed4c81ea-a9b7-43c6-a893-342cd3d4389f-kube-api-access-65brg\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:41.291822 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.291661 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jklbs\" (UniqueName: \"kubernetes.io/projected/804a9cae-91df-441a-a59d-e8c117344b78-kube-api-access-jklbs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:41.291822 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.291670 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/804a9cae-91df-441a-a59d-e8c117344b78-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:41.291822 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.291681 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/804a9cae-91df-441a-a59d-e8c117344b78-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:41.291822 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.291691 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bp4x\" (UniqueName: \"kubernetes.io/projected/a8aa349b-e04a-4726-9b2e-a910d328e7f9-kube-api-access-4bp4x\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:41.483658 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.483623 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae60c2b-2e14-416e-b552-fb9b929be559" path="/var/lib/kubelet/pods/1ae60c2b-2e14-416e-b552-fb9b929be559/volumes" Oct 29 23:51:41.484147 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:41.484133 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f99eb02d-30df-4333-b4a3-ea8b5ff576cf" path="/var/lib/kubelet/pods/f99eb02d-30df-4333-b4a3-ea8b5ff576cf/volumes" Oct 29 23:51:42.002061 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:42.002027 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw" Oct 29 23:51:42.002325 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:42.002026 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw" event={"ID":"ed4c81ea-a9b7-43c6-a893-342cd3d4389f","Type":"ContainerDied","Data":"0c28f09290f195a104ec2ba1639fb46971682203510b4c33a142d844c37d2082"} Oct 29 23:51:42.002325 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:42.002217 5057 scope.go:122] "RemoveContainer" containerID="c7ed05d5970021f0ea51b7d5fd9456213ffe5c8f73d5e1eafe864750c8607ce2" Oct 29 23:51:42.005152 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:42.004829 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" Oct 29 23:51:42.005152 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:42.004822 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2" event={"ID":"804a9cae-91df-441a-a59d-e8c117344b78","Type":"ContainerDied","Data":"6d67f2b5ad03b8e454b7811e77ba69b37c7af91ab437d2b9282e14248493b228"} Oct 29 23:51:42.007554 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:42.007530 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b" event={"ID":"a8aa349b-e04a-4726-9b2e-a910d328e7f9","Type":"ContainerDied","Data":"86e506f2c6ec24872b46d0473fce7005903b7b5e1ea93361414b51b72a7c35cf"} Oct 29 23:51:42.007667 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:42.007543 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b" Oct 29 23:51:42.013740 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:42.013720 5057 scope.go:122] "RemoveContainer" containerID="97af36718ab43b288f901767f79cf6812d6908afdcd2179dde6719f80be869aa" Oct 29 23:51:42.027761 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:42.027734 5057 scope.go:122] "RemoveContainer" containerID="baa540b9867d14046559e73916cb37c00e597b42420462eea5f7a2011f7a72a7" Oct 29 23:51:42.055110 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:42.055078 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b"] Oct 29 23:51:42.068034 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:42.068002 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3b8149-b489cfcf5-d759b"] Oct 29 23:51:42.095763 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:42.095707 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2"] Oct 29 23:51:42.099050 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:42.099022 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-bfdp2"] Oct 29 23:51:42.133526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:42.133494 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw"] Oct 29 23:51:42.136750 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:42.136719 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3b8149-7666d49bb-6mfpw"] Oct 29 23:51:43.483877 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:43.483846 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="804a9cae-91df-441a-a59d-e8c117344b78" path="/var/lib/kubelet/pods/804a9cae-91df-441a-a59d-e8c117344b78/volumes" Oct 29 23:51:43.484405 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:43.484384 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8aa349b-e04a-4726-9b2e-a910d328e7f9" path="/var/lib/kubelet/pods/a8aa349b-e04a-4726-9b2e-a910d328e7f9/volumes" Oct 29 23:51:43.484849 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:43.484835 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed4c81ea-a9b7-43c6-a893-342cd3d4389f" path="/var/lib/kubelet/pods/ed4c81ea-a9b7-43c6-a893-342cd3d4389f/volumes" Oct 29 23:51:46.584985 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:46.584933 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt"] Oct 29 23:51:46.598320 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:46.598285 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg"] Oct 29 23:51:46.598567 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:46.598511 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" podUID="23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1" containerName="maas-api" containerID="cri-o://f91c81529f6b9fe99e4812e82d93b35b41c80e7e48fd9590778a61b68172001c" gracePeriod=30 Oct 29 23:51:46.896505 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:46.896471 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" Oct 29 23:51:47.000320 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.000288 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-openshift-service-ca\") pod \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\" (UID: \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\") " Oct 29 23:51:47.000511 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.000424 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-metrics-certs\") pod \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\" (UID: \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\") " Oct 29 23:51:47.000511 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.000472 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-kube-api-access-q8s86\" (UniqueName: \"kubernetes.io/projected/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-kube-api-access-q8s86\") pod \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\" (UID: \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\") " Oct 29 23:51:47.000722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.000697 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-openshift-service-ca" pod "23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1" (UID: "23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:51:47.000781 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.000769 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-maas-api-tls\") pod \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\" (UID: \"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1\") " Oct 29 23:51:47.001382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.001359 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:47.003121 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.003087 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-maas-api-tls" pod "23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1" (UID: "23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:51:47.003265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.003123 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-metrics-certs" pod "23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1" (UID: "23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:51:47.003378 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.003356 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-kube-api-access-q8s86" pod "23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1" (UID: "23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1"). InnerVolumeSpecName "kube-api-access-q8s86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:47.052629 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.052590 5057 generic.go:356] "Generic (PLEG): container finished" podID="23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1" containerID="f91c81529f6b9fe99e4812e82d93b35b41c80e7e48fd9590778a61b68172001c" exitCode=0 Oct 29 23:51:47.052833 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.052654 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" Oct 29 23:51:47.052833 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.052676 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" event={"ID":"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1","Type":"ContainerDied","Data":"f91c81529f6b9fe99e4812e82d93b35b41c80e7e48fd9590778a61b68172001c"} Oct 29 23:51:47.052833 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.052747 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg" event={"ID":"23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1","Type":"ContainerDied","Data":"6709d4262a0236a6b62ff88cb3d5354683d45b7c88c4db19838b374a1379cfb6"} Oct 29 23:51:47.052833 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.052786 5057 scope.go:122] "RemoveContainer" containerID="f91c81529f6b9fe99e4812e82d93b35b41c80e7e48fd9590778a61b68172001c" Oct 29 23:51:47.064785 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.064760 5057 scope.go:122] "RemoveContainer" containerID="f91c81529f6b9fe99e4812e82d93b35b41c80e7e48fd9590778a61b68172001c" Oct 29 23:51:47.065106 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:51:47.065077 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91c81529f6b9fe99e4812e82d93b35b41c80e7e48fd9590778a61b68172001c\": container with ID starting with f91c81529f6b9fe99e4812e82d93b35b41c80e7e48fd9590778a61b68172001c not found: ID does not exist" containerID="f91c81529f6b9fe99e4812e82d93b35b41c80e7e48fd9590778a61b68172001c" Oct 29 23:51:47.065179 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.065119 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91c81529f6b9fe99e4812e82d93b35b41c80e7e48fd9590778a61b68172001c"} err="failed to get container status \"f91c81529f6b9fe99e4812e82d93b35b41c80e7e48fd9590778a61b68172001c\": rpc error: code = NotFound desc = could not find container \"f91c81529f6b9fe99e4812e82d93b35b41c80e7e48fd9590778a61b68172001c\": container with ID starting with f91c81529f6b9fe99e4812e82d93b35b41c80e7e48fd9590778a61b68172001c not found: ID does not exist" Oct 29 23:51:47.102427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.102387 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:47.102427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.102420 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:47.102427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.102435 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8s86\" (UniqueName: \"kubernetes.io/projected/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1-kube-api-access-q8s86\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:47.106945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.106912 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg"] Oct 29 23:51:47.110090 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.110025 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8149-6df956f8b7-jxjmg"] Oct 29 23:51:47.482684 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:47.482648 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1" path="/var/lib/kubelet/pods/23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1/volumes" Oct 29 23:51:50.712980 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:50.712941 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8"] Oct 29 23:51:51.411848 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:51.410800 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2"] Oct 29 23:51:51.439336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:51.439300 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh"] Oct 29 23:51:51.585557 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:51.585508 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt" podUID="c9adfb1e-b9a5-4f30-9bba-ef91a855ff71" containerName="payload-pre-processing" containerID="cri-o://0c02e71e7d4dde6eeef528e81e1f654a41fe2c944ad6a7a01b0722fc824a5750" gracePeriod=25 Oct 29 23:51:54.387356 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:54.387323 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6"] Oct 29 23:51:54.387845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:54.387586 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" podUID="1de79fc1-b30d-40eb-9f6d-55b1ddde3139" containerName="istio-proxy" containerID="cri-o://2a3bbad587a0352c4e8e0da33e0b5a41ed4832fb9067b754026e424ef2eb6639" gracePeriod=30 Oct 29 23:51:55.713443 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:55.713402 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8" podUID="4494b17b-7e7b-41e4-a759-0284b87f5a57" containerName="payload-pre-processing" containerID="cri-o://4e7dea9121622967d793a2dac0763d44a1e2811946cfe18e403cbe3eee9ab96e" gracePeriod=25 Oct 29 23:51:55.986278 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:55.986161 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" containerID="cri-o://e884af9ba506e29c0f0414cc733eb6e50bdcc757b1eb46ed735235d9aceb115a" gracePeriod=45 Oct 29 23:51:56.017976 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.017942 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": dial tcp 10.132.0.82:8000: connect: connection refused" Oct 29 23:51:56.062119 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.062078 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt" podUID="c9adfb1e-b9a5-4f30-9bba-ef91a855ff71" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:51:56.153742 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.153706 5057 generic.go:356] "Generic (PLEG): container finished" podID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerID="e884af9ba506e29c0f0414cc733eb6e50bdcc757b1eb46ed735235d9aceb115a" exitCode=0 Oct 29 23:51:56.153913 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.153759 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" event={"ID":"017ea90c-95e4-4199-8f90-e67ee5f80c41","Type":"ContainerDied","Data":"e884af9ba506e29c0f0414cc733eb6e50bdcc757b1eb46ed735235d9aceb115a"} Oct 29 23:51:56.309699 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.309658 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:51:56.371970 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.371914 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/017ea90c-95e4-4199-8f90-e67ee5f80c41-kube-api-access-nrr22\" (UniqueName: \"kubernetes.io/projected/017ea90c-95e4-4199-8f90-e67ee5f80c41-kube-api-access-nrr22\") pod \"017ea90c-95e4-4199-8f90-e67ee5f80c41\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " Oct 29 23:51:56.372160 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.372036 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-model-cache\") pod \"017ea90c-95e4-4199-8f90-e67ee5f80c41\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " Oct 29 23:51:56.372160 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.372075 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-dshm\" (UniqueName: \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-dshm\") pod \"017ea90c-95e4-4199-8f90-e67ee5f80c41\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " Oct 29 23:51:56.372302 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.372218 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-home\" (UniqueName: \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-home\") pod \"017ea90c-95e4-4199-8f90-e67ee5f80c41\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " Oct 29 23:51:56.372916 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.372887 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-tmp-dir\") pod \"017ea90c-95e4-4199-8f90-e67ee5f80c41\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " Oct 29 23:51:56.373076 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.372985 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/017ea90c-95e4-4199-8f90-e67ee5f80c41-tls-certs\" (UniqueName: \"kubernetes.io/secret/017ea90c-95e4-4199-8f90-e67ee5f80c41-tls-certs\") pod \"017ea90c-95e4-4199-8f90-e67ee5f80c41\" (UID: \"017ea90c-95e4-4199-8f90-e67ee5f80c41\") " Oct 29 23:51:56.374199 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.374155 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-home" pod "017ea90c-95e4-4199-8f90-e67ee5f80c41" (UID: "017ea90c-95e4-4199-8f90-e67ee5f80c41"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:56.374899 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.374849 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-model-cache" pod "017ea90c-95e4-4199-8f90-e67ee5f80c41" (UID: "017ea90c-95e4-4199-8f90-e67ee5f80c41"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:56.375100 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.375067 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-tmp-dir" pod "017ea90c-95e4-4199-8f90-e67ee5f80c41" (UID: "017ea90c-95e4-4199-8f90-e67ee5f80c41"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:56.376737 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.376705 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017ea90c-95e4-4199-8f90-e67ee5f80c41-kube-api-access-nrr22" pod "017ea90c-95e4-4199-8f90-e67ee5f80c41" (UID: "017ea90c-95e4-4199-8f90-e67ee5f80c41"). InnerVolumeSpecName "kube-api-access-nrr22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:56.376978 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.376954 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-dshm" pod "017ea90c-95e4-4199-8f90-e67ee5f80c41" (UID: "017ea90c-95e4-4199-8f90-e67ee5f80c41"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:56.377160 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.377128 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/017ea90c-95e4-4199-8f90-e67ee5f80c41-tls-certs" pod "017ea90c-95e4-4199-8f90-e67ee5f80c41" (UID: "017ea90c-95e4-4199-8f90-e67ee5f80c41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:51:56.412036 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.411984 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2" podUID="98609461-cc7d-449e-8dc9-f035a5c53d94" containerName="payload-processing" containerID="cri-o://3019edab239b304f9ea2a3907104d17c6b3dc1a22d31eec9e65e0e4f63fdf267" gracePeriod=25 Oct 29 23:51:56.439902 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.439838 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh" podUID="cb34f658-47bb-4941-bdca-53d80e885bf6" containerName="payload-pre-processing" containerID="cri-o://be1f3c5bbb82f7e9c09d46575b168ee0b77b94d6ad1b0a968b6b3367708d8159" gracePeriod=25 Oct 29 23:51:56.475000 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.474968 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-home\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.475163 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.475012 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-tmp-dir\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.475163 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.475021 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017ea90c-95e4-4199-8f90-e67ee5f80c41-tls-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.475163 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.475030 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrr22\" (UniqueName: \"kubernetes.io/projected/017ea90c-95e4-4199-8f90-e67ee5f80c41-kube-api-access-nrr22\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.475163 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.475040 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-model-cache\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.475163 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.475048 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017ea90c-95e4-4199-8f90-e67ee5f80c41-dshm\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.858244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.833775 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr"] Oct 29 23:51:56.858244 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:56.834044 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" podUID="f523466e-3408-48cc-aa67-21eec54c0eb0" containerName="maas-api" containerID="cri-o://6444cb5cfbcfb421c80865814210ae16b4bb5b1fd8af60a21f1229a71e4f3e09" gracePeriod=30 Oct 29 23:51:57.187745 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.187705 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" event={"ID":"017ea90c-95e4-4199-8f90-e67ee5f80c41","Type":"ContainerDied","Data":"c8219443fa4caf57a2e0d754f7d2f29c2b9dea1f877422403b1d2d9345f4861a"} Oct 29 23:51:57.187934 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.187761 5057 scope.go:122] "RemoveContainer" containerID="e884af9ba506e29c0f0414cc733eb6e50bdcc757b1eb46ed735235d9aceb115a" Oct 29 23:51:57.187934 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.187924 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc" Oct 29 23:51:57.190336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.190307 5057 generic.go:356] "Generic (PLEG): container finished" podID="f523466e-3408-48cc-aa67-21eec54c0eb0" containerID="6444cb5cfbcfb421c80865814210ae16b4bb5b1fd8af60a21f1229a71e4f3e09" exitCode=0 Oct 29 23:51:57.190462 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.190360 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" event={"ID":"f523466e-3408-48cc-aa67-21eec54c0eb0","Type":"ContainerDied","Data":"6444cb5cfbcfb421c80865814210ae16b4bb5b1fd8af60a21f1229a71e4f3e09"} Oct 29 23:51:57.247957 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.247922 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" Oct 29 23:51:57.260239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.257156 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc"] Oct 29 23:51:57.260998 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.260966 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/distinct-2-w6-1ee571-kserve-68d65c66cb-fhrfc"] Oct 29 23:51:57.318226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.307669 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f523466e-3408-48cc-aa67-21eec54c0eb0-kube-api-access-9nkgd\" (UniqueName: \"kubernetes.io/projected/f523466e-3408-48cc-aa67-21eec54c0eb0-kube-api-access-9nkgd\") pod \"f523466e-3408-48cc-aa67-21eec54c0eb0\" (UID: \"f523466e-3408-48cc-aa67-21eec54c0eb0\") " Oct 29 23:51:57.331219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.327399 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f523466e-3408-48cc-aa67-21eec54c0eb0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f523466e-3408-48cc-aa67-21eec54c0eb0-openshift-service-ca\") pod \"f523466e-3408-48cc-aa67-21eec54c0eb0\" (UID: \"f523466e-3408-48cc-aa67-21eec54c0eb0\") " Oct 29 23:51:57.331219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.327528 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f523466e-3408-48cc-aa67-21eec54c0eb0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f523466e-3408-48cc-aa67-21eec54c0eb0-maas-api-tls\") pod \"f523466e-3408-48cc-aa67-21eec54c0eb0\" (UID: \"f523466e-3408-48cc-aa67-21eec54c0eb0\") " Oct 29 23:51:57.331219 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.327695 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f523466e-3408-48cc-aa67-21eec54c0eb0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f523466e-3408-48cc-aa67-21eec54c0eb0-metrics-certs\") pod \"f523466e-3408-48cc-aa67-21eec54c0eb0\" (UID: \"f523466e-3408-48cc-aa67-21eec54c0eb0\") " Oct 29 23:51:57.337405 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.337371 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f523466e-3408-48cc-aa67-21eec54c0eb0-openshift-service-ca" pod "f523466e-3408-48cc-aa67-21eec54c0eb0" (UID: "f523466e-3408-48cc-aa67-21eec54c0eb0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:51:57.377361 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.342331 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f523466e-3408-48cc-aa67-21eec54c0eb0-kube-api-access-9nkgd" pod "f523466e-3408-48cc-aa67-21eec54c0eb0" (UID: "f523466e-3408-48cc-aa67-21eec54c0eb0"). InnerVolumeSpecName "kube-api-access-9nkgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:57.389407 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.389346 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f523466e-3408-48cc-aa67-21eec54c0eb0-metrics-certs" pod "f523466e-3408-48cc-aa67-21eec54c0eb0" (UID: "f523466e-3408-48cc-aa67-21eec54c0eb0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:51:57.390059 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.390015 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f523466e-3408-48cc-aa67-21eec54c0eb0-maas-api-tls" pod "f523466e-3408-48cc-aa67-21eec54c0eb0" (UID: "f523466e-3408-48cc-aa67-21eec54c0eb0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:51:57.434333 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.434059 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f523466e-3408-48cc-aa67-21eec54c0eb0-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:57.434333 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.434099 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f523466e-3408-48cc-aa67-21eec54c0eb0-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:57.434333 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.434114 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nkgd\" (UniqueName: \"kubernetes.io/projected/f523466e-3408-48cc-aa67-21eec54c0eb0-kube-api-access-9nkgd\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:57.434333 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.434128 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f523466e-3408-48cc-aa67-21eec54c0eb0-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:57.483228 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:57.483121 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" path="/var/lib/kubelet/pods/017ea90c-95e4-4199-8f90-e67ee5f80c41/volumes" Oct 29 23:51:58.201067 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:58.201017 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" event={"ID":"f523466e-3408-48cc-aa67-21eec54c0eb0","Type":"ContainerDied","Data":"df0144c7b108fba061a88401a6455c3f46233d5518f74b23aa17e53dd9d82fbc"} Oct 29 23:51:58.201580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:58.201098 5057 scope.go:122] "RemoveContainer" containerID="6444cb5cfbcfb421c80865814210ae16b4bb5b1fd8af60a21f1229a71e4f3e09" Oct 29 23:51:58.201580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:58.201232 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr" Oct 29 23:51:58.253808 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:58.253780 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr"] Oct 29 23:51:58.266355 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:58.266327 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e1769a-6888dcd8c5-svxrr"] Oct 29 23:51:59.166987 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.166951 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh" podUID="cb34f658-47bb-4941-bdca-53d80e885bf6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:51:59.167305 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.167281 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2" podUID="98609461-cc7d-449e-8dc9-f035a5c53d94" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:51:59.482771 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.482731 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f523466e-3408-48cc-aa67-21eec54c0eb0" path="/var/lib/kubelet/pods/f523466e-3408-48cc-aa67-21eec54c0eb0/volumes" Oct 29 23:51:59.689515 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.689488 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:51:59.783319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.783284 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-credential-socket\") pod \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " Oct 29 23:51:59.783886 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.783866 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-kube-api-access-x69nx\" (UniqueName: \"kubernetes.io/projected/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-kube-api-access-x69nx\") pod \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " Oct 29 23:51:59.784536 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.784513 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-workload-socket\") pod \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " Oct 29 23:51:59.784950 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.784933 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istiod-ca-cert\") pod \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " Oct 29 23:51:59.785442 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.785416 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-workload-certs\") pod \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " Oct 29 23:51:59.785880 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.785863 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-podinfo\") pod \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " Oct 29 23:51:59.787311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.783776 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-credential-socket" pod "1de79fc1-b30d-40eb-9f6d-55b1ddde3139" (UID: "1de79fc1-b30d-40eb-9f6d-55b1ddde3139"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:59.787311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.784872 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-workload-socket" pod "1de79fc1-b30d-40eb-9f6d-55b1ddde3139" (UID: "1de79fc1-b30d-40eb-9f6d-55b1ddde3139"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:59.787311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.785369 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istiod-ca-cert" pod "1de79fc1-b30d-40eb-9f6d-55b1ddde3139" (UID: "1de79fc1-b30d-40eb-9f6d-55b1ddde3139"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:51:59.787311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.785784 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-workload-certs" pod "1de79fc1-b30d-40eb-9f6d-55b1ddde3139" (UID: "1de79fc1-b30d-40eb-9f6d-55b1ddde3139"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:59.787311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.786393 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-data\") pod \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " Oct 29 23:51:59.787311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.786472 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-token\" (UniqueName: \"kubernetes.io/projected/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-token\") pod \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " Oct 29 23:51:59.787311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.786517 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-envoy\") pod \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\" (UID: \"1de79fc1-b30d-40eb-9f6d-55b1ddde3139\") " Oct 29 23:51:59.787311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.786886 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-data" pod "1de79fc1-b30d-40eb-9f6d-55b1ddde3139" (UID: "1de79fc1-b30d-40eb-9f6d-55b1ddde3139"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:59.787311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.787217 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:59.787311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.787236 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:59.787311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.787250 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:59.787311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.787264 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:59.787311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.787278 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:59.788837 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.788804 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-kube-api-access-x69nx" pod "1de79fc1-b30d-40eb-9f6d-55b1ddde3139" (UID: "1de79fc1-b30d-40eb-9f6d-55b1ddde3139"). InnerVolumeSpecName "kube-api-access-x69nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:59.791035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.791002 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-podinfo" pod "1de79fc1-b30d-40eb-9f6d-55b1ddde3139" (UID: "1de79fc1-b30d-40eb-9f6d-55b1ddde3139"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:51:59.791262 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.791226 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-envoy" pod "1de79fc1-b30d-40eb-9f6d-55b1ddde3139" (UID: "1de79fc1-b30d-40eb-9f6d-55b1ddde3139"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:59.791696 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.791663 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-token" pod "1de79fc1-b30d-40eb-9f6d-55b1ddde3139" (UID: "1de79fc1-b30d-40eb-9f6d-55b1ddde3139"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:59.888914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.888883 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:59.888914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.888911 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:59.888914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.888922 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x69nx\" (UniqueName: \"kubernetes.io/projected/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-kube-api-access-x69nx\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:51:59.889154 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:51:59.888937 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1de79fc1-b30d-40eb-9f6d-55b1ddde3139-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:00.217626 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:00.217593 5057 generic.go:356] "Generic (PLEG): container finished" podID="1de79fc1-b30d-40eb-9f6d-55b1ddde3139" containerID="2a3bbad587a0352c4e8e0da33e0b5a41ed4832fb9067b754026e424ef2eb6639" exitCode=0 Oct 29 23:52:00.217862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:00.217647 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" Oct 29 23:52:00.217862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:00.217681 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" event={"ID":"1de79fc1-b30d-40eb-9f6d-55b1ddde3139","Type":"ContainerDied","Data":"2a3bbad587a0352c4e8e0da33e0b5a41ed4832fb9067b754026e424ef2eb6639"} Oct 29 23:52:00.217862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:00.217749 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6" event={"ID":"1de79fc1-b30d-40eb-9f6d-55b1ddde3139","Type":"ContainerDied","Data":"0de7e1e05641f9b1e0294bd7ff9d7f5c46ecc55b1b91f51ae3eb80a21406c188"} Oct 29 23:52:00.217862 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:00.217778 5057 scope.go:122] "RemoveContainer" containerID="2a3bbad587a0352c4e8e0da33e0b5a41ed4832fb9067b754026e424ef2eb6639" Oct 29 23:52:00.229516 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:00.229487 5057 scope.go:122] "RemoveContainer" containerID="2a3bbad587a0352c4e8e0da33e0b5a41ed4832fb9067b754026e424ef2eb6639" Oct 29 23:52:00.229849 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:52:00.229809 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a3bbad587a0352c4e8e0da33e0b5a41ed4832fb9067b754026e424ef2eb6639\": container with ID starting with 2a3bbad587a0352c4e8e0da33e0b5a41ed4832fb9067b754026e424ef2eb6639 not found: ID does not exist" containerID="2a3bbad587a0352c4e8e0da33e0b5a41ed4832fb9067b754026e424ef2eb6639" Oct 29 23:52:00.229908 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:00.229857 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a3bbad587a0352c4e8e0da33e0b5a41ed4832fb9067b754026e424ef2eb6639"} err="failed to get container status \"2a3bbad587a0352c4e8e0da33e0b5a41ed4832fb9067b754026e424ef2eb6639\": rpc error: code = NotFound desc = could not find container \"2a3bbad587a0352c4e8e0da33e0b5a41ed4832fb9067b754026e424ef2eb6639\": container with ID starting with 2a3bbad587a0352c4e8e0da33e0b5a41ed4832fb9067b754026e424ef2eb6639 not found: ID does not exist" Oct 29 23:52:00.272613 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:00.272579 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6"] Oct 29 23:52:00.284139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:00.284105 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-k8bx6"] Oct 29 23:52:01.481834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:01.481796 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1de79fc1-b30d-40eb-9f6d-55b1ddde3139" path="/var/lib/kubelet/pods/1de79fc1-b30d-40eb-9f6d-55b1ddde3139/volumes" Oct 29 23:52:01.535691 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:01.535647 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8" podUID="4494b17b-7e7b-41e4-a759-0284b87f5a57" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:52:03.917539 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:52:03.917473 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:52:06.061426 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:06.061390 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt" podUID="c9adfb1e-b9a5-4f30-9bba-ef91a855ff71" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:52:09.167253 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:09.167176 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh" podUID="cb34f658-47bb-4941-bdca-53d80e885bf6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:52:09.167253 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:09.167237 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2" podUID="98609461-cc7d-449e-8dc9-f035a5c53d94" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:52:11.535428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:11.535387 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8" podUID="4494b17b-7e7b-41e4-a759-0284b87f5a57" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:52:11.905311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:11.905279 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt" Oct 29 23:52:12.086282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:12.086240 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9adfb1e-b9a5-4f30-9bba-ef91a855ff71-kube-api-access-grddx\" (UniqueName: \"kubernetes.io/projected/c9adfb1e-b9a5-4f30-9bba-ef91a855ff71-kube-api-access-grddx\") pod \"c9adfb1e-b9a5-4f30-9bba-ef91a855ff71\" (UID: \"c9adfb1e-b9a5-4f30-9bba-ef91a855ff71\") " Oct 29 23:52:12.086479 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:12.086448 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9adfb1e-b9a5-4f30-9bba-ef91a855ff71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9adfb1e-b9a5-4f30-9bba-ef91a855ff71-plugins-config-volume\") pod \"c9adfb1e-b9a5-4f30-9bba-ef91a855ff71\" (UID: \"c9adfb1e-b9a5-4f30-9bba-ef91a855ff71\") " Oct 29 23:52:12.086831 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:12.086804 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9adfb1e-b9a5-4f30-9bba-ef91a855ff71-plugins-config-volume" pod "c9adfb1e-b9a5-4f30-9bba-ef91a855ff71" (UID: "c9adfb1e-b9a5-4f30-9bba-ef91a855ff71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:12.086977 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:12.086963 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9adfb1e-b9a5-4f30-9bba-ef91a855ff71-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:12.088470 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:12.088448 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9adfb1e-b9a5-4f30-9bba-ef91a855ff71-kube-api-access-grddx" pod "c9adfb1e-b9a5-4f30-9bba-ef91a855ff71" (UID: "c9adfb1e-b9a5-4f30-9bba-ef91a855ff71"). InnerVolumeSpecName "kube-api-access-grddx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:12.188267 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:12.188227 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grddx\" (UniqueName: \"kubernetes.io/projected/c9adfb1e-b9a5-4f30-9bba-ef91a855ff71-kube-api-access-grddx\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:12.323235 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:12.323180 5057 generic.go:356] "Generic (PLEG): container finished" podID="c9adfb1e-b9a5-4f30-9bba-ef91a855ff71" containerID="0c02e71e7d4dde6eeef528e81e1f654a41fe2c944ad6a7a01b0722fc824a5750" exitCode=0 Oct 29 23:52:12.323447 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:12.323257 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt" Oct 29 23:52:12.323447 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:12.323287 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt" event={"ID":"c9adfb1e-b9a5-4f30-9bba-ef91a855ff71","Type":"ContainerDied","Data":"0c02e71e7d4dde6eeef528e81e1f654a41fe2c944ad6a7a01b0722fc824a5750"} Oct 29 23:52:12.323447 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:12.323386 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt" event={"ID":"c9adfb1e-b9a5-4f30-9bba-ef91a855ff71","Type":"ContainerDied","Data":"8146b7f497a47acfef1903da801ba2c3a5d2cdc37a698f7694b9031da5c38a08"} Oct 29 23:52:12.323447 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:12.323418 5057 scope.go:122] "RemoveContainer" containerID="0c02e71e7d4dde6eeef528e81e1f654a41fe2c944ad6a7a01b0722fc824a5750" Oct 29 23:52:12.336012 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:12.335990 5057 scope.go:122] "RemoveContainer" containerID="0c02e71e7d4dde6eeef528e81e1f654a41fe2c944ad6a7a01b0722fc824a5750" Oct 29 23:52:12.336369 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:52:12.336346 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c02e71e7d4dde6eeef528e81e1f654a41fe2c944ad6a7a01b0722fc824a5750\": container with ID starting with 0c02e71e7d4dde6eeef528e81e1f654a41fe2c944ad6a7a01b0722fc824a5750 not found: ID does not exist" containerID="0c02e71e7d4dde6eeef528e81e1f654a41fe2c944ad6a7a01b0722fc824a5750" Oct 29 23:52:12.336453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:12.336375 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c02e71e7d4dde6eeef528e81e1f654a41fe2c944ad6a7a01b0722fc824a5750"} err="failed to get container status \"0c02e71e7d4dde6eeef528e81e1f654a41fe2c944ad6a7a01b0722fc824a5750\": rpc error: code = NotFound desc = could not find container \"0c02e71e7d4dde6eeef528e81e1f654a41fe2c944ad6a7a01b0722fc824a5750\": container with ID starting with 0c02e71e7d4dde6eeef528e81e1f654a41fe2c944ad6a7a01b0722fc824a5750 not found: ID does not exist" Oct 29 23:52:12.376576 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:12.376541 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt"] Oct 29 23:52:12.389806 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:12.389777 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-1ee571-7dd944c97f-cr2rt"] Oct 29 23:52:13.483942 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:13.483910 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9adfb1e-b9a5-4f30-9bba-ef91a855ff71" path="/var/lib/kubelet/pods/c9adfb1e-b9a5-4f30-9bba-ef91a855ff71/volumes" Oct 29 23:52:15.066784 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:15.066747 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk"] Oct 29 23:52:15.067746 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:15.067706 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" podUID="5c575af2-2a8a-4e83-adcc-7d6578c53235" containerName="istio-proxy" containerID="cri-o://cceaaf0130b1dac747c78237db3470ead33ef2647084c4297051d73fe08c043e" gracePeriod=30 Oct 29 23:52:16.021243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.021214 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8" Oct 29 23:52:16.073074 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.073001 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4494b17b-7e7b-41e4-a759-0284b87f5a57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4494b17b-7e7b-41e4-a759-0284b87f5a57-plugins-config-volume\") pod \"4494b17b-7e7b-41e4-a759-0284b87f5a57\" (UID: \"4494b17b-7e7b-41e4-a759-0284b87f5a57\") " Oct 29 23:52:16.073583 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.073347 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4494b17b-7e7b-41e4-a759-0284b87f5a57-kube-api-access-bjqs5\" (UniqueName: \"kubernetes.io/projected/4494b17b-7e7b-41e4-a759-0284b87f5a57-kube-api-access-bjqs5\") pod \"4494b17b-7e7b-41e4-a759-0284b87f5a57\" (UID: \"4494b17b-7e7b-41e4-a759-0284b87f5a57\") " Oct 29 23:52:16.073583 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.073455 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4494b17b-7e7b-41e4-a759-0284b87f5a57-plugins-config-volume" pod "4494b17b-7e7b-41e4-a759-0284b87f5a57" (UID: "4494b17b-7e7b-41e4-a759-0284b87f5a57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:16.073930 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.073907 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4494b17b-7e7b-41e4-a759-0284b87f5a57-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:16.075667 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.075629 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4494b17b-7e7b-41e4-a759-0284b87f5a57-kube-api-access-bjqs5" pod "4494b17b-7e7b-41e4-a759-0284b87f5a57" (UID: "4494b17b-7e7b-41e4-a759-0284b87f5a57"). InnerVolumeSpecName "kube-api-access-bjqs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:16.174747 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.174712 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjqs5\" (UniqueName: \"kubernetes.io/projected/4494b17b-7e7b-41e4-a759-0284b87f5a57-kube-api-access-bjqs5\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:16.372349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.372250 5057 generic.go:356] "Generic (PLEG): container finished" podID="4494b17b-7e7b-41e4-a759-0284b87f5a57" containerID="4e7dea9121622967d793a2dac0763d44a1e2811946cfe18e403cbe3eee9ab96e" exitCode=0 Oct 29 23:52:16.372349 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.372320 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8" event={"ID":"4494b17b-7e7b-41e4-a759-0284b87f5a57","Type":"ContainerDied","Data":"4e7dea9121622967d793a2dac0763d44a1e2811946cfe18e403cbe3eee9ab96e"} Oct 29 23:52:16.372580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.372371 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8" event={"ID":"4494b17b-7e7b-41e4-a759-0284b87f5a57","Type":"ContainerDied","Data":"2a0dfd54adffb546021a584c7f767eec6703f695b065a1f5cd6c699e14069e8d"} Oct 29 23:52:16.372580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.372399 5057 scope.go:122] "RemoveContainer" containerID="4e7dea9121622967d793a2dac0763d44a1e2811946cfe18e403cbe3eee9ab96e" Oct 29 23:52:16.372931 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.372895 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8" Oct 29 23:52:16.387451 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.387416 5057 scope.go:122] "RemoveContainer" containerID="4e7dea9121622967d793a2dac0763d44a1e2811946cfe18e403cbe3eee9ab96e" Oct 29 23:52:16.387760 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:52:16.387734 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e7dea9121622967d793a2dac0763d44a1e2811946cfe18e403cbe3eee9ab96e\": container with ID starting with 4e7dea9121622967d793a2dac0763d44a1e2811946cfe18e403cbe3eee9ab96e not found: ID does not exist" containerID="4e7dea9121622967d793a2dac0763d44a1e2811946cfe18e403cbe3eee9ab96e" Oct 29 23:52:16.387833 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.387772 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e7dea9121622967d793a2dac0763d44a1e2811946cfe18e403cbe3eee9ab96e"} err="failed to get container status \"4e7dea9121622967d793a2dac0763d44a1e2811946cfe18e403cbe3eee9ab96e\": rpc error: code = NotFound desc = could not find container \"4e7dea9121622967d793a2dac0763d44a1e2811946cfe18e403cbe3eee9ab96e\": container with ID starting with 4e7dea9121622967d793a2dac0763d44a1e2811946cfe18e403cbe3eee9ab96e not found: ID does not exist" Oct 29 23:52:16.431555 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.431521 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8"] Oct 29 23:52:16.431706 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.431566 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-37498a-676bb9c5ff-pkbs8"] Oct 29 23:52:16.702673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.702644 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2" Oct 29 23:52:16.783278 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.783247 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98609461-cc7d-449e-8dc9-f035a5c53d94-kube-api-access-dxmxn\" (UniqueName: \"kubernetes.io/projected/98609461-cc7d-449e-8dc9-f035a5c53d94-kube-api-access-dxmxn\") pod \"98609461-cc7d-449e-8dc9-f035a5c53d94\" (UID: \"98609461-cc7d-449e-8dc9-f035a5c53d94\") " Oct 29 23:52:16.783502 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.783484 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98609461-cc7d-449e-8dc9-f035a5c53d94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98609461-cc7d-449e-8dc9-f035a5c53d94-plugins-config-volume\") pod \"98609461-cc7d-449e-8dc9-f035a5c53d94\" (UID: \"98609461-cc7d-449e-8dc9-f035a5c53d94\") " Oct 29 23:52:16.784151 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.784119 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98609461-cc7d-449e-8dc9-f035a5c53d94-plugins-config-volume" pod "98609461-cc7d-449e-8dc9-f035a5c53d94" (UID: "98609461-cc7d-449e-8dc9-f035a5c53d94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:16.785680 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.785657 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98609461-cc7d-449e-8dc9-f035a5c53d94-kube-api-access-dxmxn" pod "98609461-cc7d-449e-8dc9-f035a5c53d94" (UID: "98609461-cc7d-449e-8dc9-f035a5c53d94"). InnerVolumeSpecName "kube-api-access-dxmxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:16.819403 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.819372 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh" Oct 29 23:52:16.885269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.885236 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb34f658-47bb-4941-bdca-53d80e885bf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb34f658-47bb-4941-bdca-53d80e885bf6-plugins-config-volume\") pod \"cb34f658-47bb-4941-bdca-53d80e885bf6\" (UID: \"cb34f658-47bb-4941-bdca-53d80e885bf6\") " Oct 29 23:52:16.885526 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.885506 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb34f658-47bb-4941-bdca-53d80e885bf6-kube-api-access-jp6d4\" (UniqueName: \"kubernetes.io/projected/cb34f658-47bb-4941-bdca-53d80e885bf6-kube-api-access-jp6d4\") pod \"cb34f658-47bb-4941-bdca-53d80e885bf6\" (UID: \"cb34f658-47bb-4941-bdca-53d80e885bf6\") " Oct 29 23:52:16.885685 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.885663 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb34f658-47bb-4941-bdca-53d80e885bf6-plugins-config-volume" pod "cb34f658-47bb-4941-bdca-53d80e885bf6" (UID: "cb34f658-47bb-4941-bdca-53d80e885bf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:16.885975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.885955 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98609461-cc7d-449e-8dc9-f035a5c53d94-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:16.885975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.885973 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxmxn\" (UniqueName: \"kubernetes.io/projected/98609461-cc7d-449e-8dc9-f035a5c53d94-kube-api-access-dxmxn\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:16.886148 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.885984 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb34f658-47bb-4941-bdca-53d80e885bf6-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:16.887754 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.887728 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb34f658-47bb-4941-bdca-53d80e885bf6-kube-api-access-jp6d4" pod "cb34f658-47bb-4941-bdca-53d80e885bf6" (UID: "cb34f658-47bb-4941-bdca-53d80e885bf6"). InnerVolumeSpecName "kube-api-access-jp6d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:16.987013 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:16.986931 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp6d4\" (UniqueName: \"kubernetes.io/projected/cb34f658-47bb-4941-bdca-53d80e885bf6-kube-api-access-jp6d4\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:17.380361 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:17.380281 5057 generic.go:356] "Generic (PLEG): container finished" podID="98609461-cc7d-449e-8dc9-f035a5c53d94" containerID="3019edab239b304f9ea2a3907104d17c6b3dc1a22d31eec9e65e0e4f63fdf267" exitCode=0 Oct 29 23:52:17.380361 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:17.380333 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2" Oct 29 23:52:17.380899 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:17.380355 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2" event={"ID":"98609461-cc7d-449e-8dc9-f035a5c53d94","Type":"ContainerDied","Data":"3019edab239b304f9ea2a3907104d17c6b3dc1a22d31eec9e65e0e4f63fdf267"} Oct 29 23:52:17.380899 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:17.380410 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2" event={"ID":"98609461-cc7d-449e-8dc9-f035a5c53d94","Type":"ContainerDied","Data":"875a05f7ca405ac8aae022fca7b40768d2f3bbb9d0fb25a151eda6a7bfe66009"} Oct 29 23:52:17.380899 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:17.380432 5057 scope.go:122] "RemoveContainer" containerID="3019edab239b304f9ea2a3907104d17c6b3dc1a22d31eec9e65e0e4f63fdf267" Oct 29 23:52:17.381945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:17.381922 5057 generic.go:356] "Generic (PLEG): container finished" podID="cb34f658-47bb-4941-bdca-53d80e885bf6" containerID="be1f3c5bbb82f7e9c09d46575b168ee0b77b94d6ad1b0a968b6b3367708d8159" exitCode=0 Oct 29 23:52:17.382055 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:17.381973 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh" Oct 29 23:52:17.382055 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:17.382017 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh" event={"ID":"cb34f658-47bb-4941-bdca-53d80e885bf6","Type":"ContainerDied","Data":"be1f3c5bbb82f7e9c09d46575b168ee0b77b94d6ad1b0a968b6b3367708d8159"} Oct 29 23:52:17.382167 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:17.382062 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh" event={"ID":"cb34f658-47bb-4941-bdca-53d80e885bf6","Type":"ContainerDied","Data":"83e7983ab69c02112821d7da80789695cb1728a71a463235e2a1f5b423fb0488"} Oct 29 23:52:17.398728 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:17.398702 5057 scope.go:122] "RemoveContainer" containerID="3019edab239b304f9ea2a3907104d17c6b3dc1a22d31eec9e65e0e4f63fdf267" Oct 29 23:52:17.399254 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:52:17.399226 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3019edab239b304f9ea2a3907104d17c6b3dc1a22d31eec9e65e0e4f63fdf267\": container with ID starting with 3019edab239b304f9ea2a3907104d17c6b3dc1a22d31eec9e65e0e4f63fdf267 not found: ID does not exist" containerID="3019edab239b304f9ea2a3907104d17c6b3dc1a22d31eec9e65e0e4f63fdf267" Oct 29 23:52:17.399387 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:17.399270 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3019edab239b304f9ea2a3907104d17c6b3dc1a22d31eec9e65e0e4f63fdf267"} err="failed to get container status \"3019edab239b304f9ea2a3907104d17c6b3dc1a22d31eec9e65e0e4f63fdf267\": rpc error: code = NotFound desc = could not find container \"3019edab239b304f9ea2a3907104d17c6b3dc1a22d31eec9e65e0e4f63fdf267\": container with ID starting with 3019edab239b304f9ea2a3907104d17c6b3dc1a22d31eec9e65e0e4f63fdf267 not found: ID does not exist" Oct 29 23:52:17.399387 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:17.399295 5057 scope.go:122] "RemoveContainer" containerID="be1f3c5bbb82f7e9c09d46575b168ee0b77b94d6ad1b0a968b6b3367708d8159" Oct 29 23:52:17.418737 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:17.418716 5057 scope.go:122] "RemoveContainer" containerID="be1f3c5bbb82f7e9c09d46575b168ee0b77b94d6ad1b0a968b6b3367708d8159" Oct 29 23:52:17.419143 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:52:17.419101 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be1f3c5bbb82f7e9c09d46575b168ee0b77b94d6ad1b0a968b6b3367708d8159\": container with ID starting with be1f3c5bbb82f7e9c09d46575b168ee0b77b94d6ad1b0a968b6b3367708d8159 not found: ID does not exist" containerID="be1f3c5bbb82f7e9c09d46575b168ee0b77b94d6ad1b0a968b6b3367708d8159" Oct 29 23:52:17.419256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:17.419162 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be1f3c5bbb82f7e9c09d46575b168ee0b77b94d6ad1b0a968b6b3367708d8159"} err="failed to get container status \"be1f3c5bbb82f7e9c09d46575b168ee0b77b94d6ad1b0a968b6b3367708d8159\": rpc error: code = NotFound desc = could not find container \"be1f3c5bbb82f7e9c09d46575b168ee0b77b94d6ad1b0a968b6b3367708d8159\": container with ID starting with be1f3c5bbb82f7e9c09d46575b168ee0b77b94d6ad1b0a968b6b3367708d8159 not found: ID does not exist" Oct 29 23:52:17.453202 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:17.453164 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh"] Oct 29 23:52:17.465078 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:17.465049 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e1769a-7b5bcb6d8c-r4fbh"] Oct 29 23:52:17.483631 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:17.483597 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4494b17b-7e7b-41e4-a759-0284b87f5a57" path="/var/lib/kubelet/pods/4494b17b-7e7b-41e4-a759-0284b87f5a57/volumes" Oct 29 23:52:17.484098 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:17.484074 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb34f658-47bb-4941-bdca-53d80e885bf6" path="/var/lib/kubelet/pods/cb34f658-47bb-4941-bdca-53d80e885bf6/volumes" Oct 29 23:52:17.493436 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:17.493406 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2"] Oct 29 23:52:17.496489 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:17.496465 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e1769a-5c88bc65c8-44nd2"] Oct 29 23:52:19.482170 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:19.482136 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98609461-cc7d-449e-8dc9-f035a5c53d94" path="/var/lib/kubelet/pods/98609461-cc7d-449e-8dc9-f035a5c53d94/volumes" Oct 29 23:52:20.568238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.568026 5057 generic.go:356] "Generic (PLEG): container finished" podID="5c575af2-2a8a-4e83-adcc-7d6578c53235" containerID="cceaaf0130b1dac747c78237db3470ead33ef2647084c4297051d73fe08c043e" exitCode=0 Oct 29 23:52:20.568238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.568096 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" event={"ID":"5c575af2-2a8a-4e83-adcc-7d6578c53235","Type":"ContainerDied","Data":"cceaaf0130b1dac747c78237db3470ead33ef2647084c4297051d73fe08c043e"} Oct 29 23:52:20.568238 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.568160 5057 scope.go:122] "RemoveContainer" containerID="c0db97b4b8cac18eb5f7eca8e2f9046c924cd55e37f6ee5d7299ba0c009eb3d9" Oct 29 23:52:20.605157 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.605135 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:52:20.706735 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.706690 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-credential-socket\") pod \"5c575af2-2a8a-4e83-adcc-7d6578c53235\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " Oct 29 23:52:20.706944 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.706814 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-podinfo\") pod \"5c575af2-2a8a-4e83-adcc-7d6578c53235\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " Oct 29 23:52:20.706944 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.706850 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-workload-socket\") pod \"5c575af2-2a8a-4e83-adcc-7d6578c53235\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " Oct 29 23:52:20.706944 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.706920 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-data\") pod \"5c575af2-2a8a-4e83-adcc-7d6578c53235\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " Oct 29 23:52:20.707136 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.706956 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-envoy\") pod \"5c575af2-2a8a-4e83-adcc-7d6578c53235\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " Oct 29 23:52:20.707136 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.707049 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-credential-socket" pod "5c575af2-2a8a-4e83-adcc-7d6578c53235" (UID: "5c575af2-2a8a-4e83-adcc-7d6578c53235"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:20.707136 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.707060 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c575af2-2a8a-4e83-adcc-7d6578c53235-kube-api-access-d9wmn\" (UniqueName: \"kubernetes.io/projected/5c575af2-2a8a-4e83-adcc-7d6578c53235-kube-api-access-d9wmn\") pod \"5c575af2-2a8a-4e83-adcc-7d6578c53235\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " Oct 29 23:52:20.707317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.707272 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c575af2-2a8a-4e83-adcc-7d6578c53235-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c575af2-2a8a-4e83-adcc-7d6578c53235-istiod-ca-cert\") pod \"5c575af2-2a8a-4e83-adcc-7d6578c53235\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " Oct 29 23:52:20.707395 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.707323 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-token\" (UniqueName: \"kubernetes.io/projected/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-token\") pod \"5c575af2-2a8a-4e83-adcc-7d6578c53235\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " Oct 29 23:52:20.707395 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.707358 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-workload-certs\") pod \"5c575af2-2a8a-4e83-adcc-7d6578c53235\" (UID: \"5c575af2-2a8a-4e83-adcc-7d6578c53235\") " Oct 29 23:52:20.708389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.707760 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c575af2-2a8a-4e83-adcc-7d6578c53235-istiod-ca-cert" pod "5c575af2-2a8a-4e83-adcc-7d6578c53235" (UID: "5c575af2-2a8a-4e83-adcc-7d6578c53235"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:20.708389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.707970 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-data" pod "5c575af2-2a8a-4e83-adcc-7d6578c53235" (UID: "5c575af2-2a8a-4e83-adcc-7d6578c53235"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:20.708389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.708163 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-workload-certs" pod "5c575af2-2a8a-4e83-adcc-7d6578c53235" (UID: "5c575af2-2a8a-4e83-adcc-7d6578c53235"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:20.708389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.708243 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-workload-socket" pod "5c575af2-2a8a-4e83-adcc-7d6578c53235" (UID: "5c575af2-2a8a-4e83-adcc-7d6578c53235"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:20.708389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.708309 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:20.708389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.708329 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c575af2-2a8a-4e83-adcc-7d6578c53235-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:20.708389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.708343 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:20.708389 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.708356 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:20.709946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.709685 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-podinfo" pod "5c575af2-2a8a-4e83-adcc-7d6578c53235" (UID: "5c575af2-2a8a-4e83-adcc-7d6578c53235"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:52:20.710567 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.710536 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c575af2-2a8a-4e83-adcc-7d6578c53235-kube-api-access-d9wmn" pod "5c575af2-2a8a-4e83-adcc-7d6578c53235" (UID: "5c575af2-2a8a-4e83-adcc-7d6578c53235"). InnerVolumeSpecName "kube-api-access-d9wmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:20.710779 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.710764 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-token" pod "5c575af2-2a8a-4e83-adcc-7d6578c53235" (UID: "5c575af2-2a8a-4e83-adcc-7d6578c53235"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:20.711415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.711399 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-envoy" pod "5c575af2-2a8a-4e83-adcc-7d6578c53235" (UID: "5c575af2-2a8a-4e83-adcc-7d6578c53235"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:20.809472 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.809436 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:20.809472 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.809465 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:20.809472 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.809474 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:20.809707 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.809485 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9wmn\" (UniqueName: \"kubernetes.io/projected/5c575af2-2a8a-4e83-adcc-7d6578c53235-kube-api-access-d9wmn\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:20.809707 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.809494 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c575af2-2a8a-4e83-adcc-7d6578c53235-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:20.827429 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:20.827356 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:52:21.578005 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:21.577962 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" event={"ID":"5c575af2-2a8a-4e83-adcc-7d6578c53235","Type":"ContainerDied","Data":"5b88c04b59a186a3fa38f82bb6ff617bf0df5f1ddb7e4094324fde5fc1280756"} Oct 29 23:52:21.578504 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:21.578036 5057 scope.go:122] "RemoveContainer" containerID="cceaaf0130b1dac747c78237db3470ead33ef2647084c4297051d73fe08c043e" Oct 29 23:52:21.578504 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:21.577969 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk" Oct 29 23:52:21.683753 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:21.683715 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk"] Oct 29 23:52:21.688401 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:21.688361 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e1769a-openshift-default-596bdb8769-h4kwk"] Oct 29 23:52:23.481840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:23.481801 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c575af2-2a8a-4e83-adcc-7d6578c53235" path="/var/lib/kubelet/pods/5c575af2-2a8a-4e83-adcc-7d6578c53235/volumes" Oct 29 23:52:26.053858 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:26.053825 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw"] Oct 29 23:52:28.555873 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:28.555837 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr"] Oct 29 23:52:28.556755 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:28.556719 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" podUID="a28590ff-f920-4993-b5f2-d8afb67b3685" containerName="istio-proxy" containerID="cri-o://efa1d67631aad66d2812bd3bfb7c2c4f3a989a84b1b89c59701e81985b7cfd04" gracePeriod=30 Oct 29 23:52:31.054582 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:31.054515 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw" podUID="41da29c2-0216-4adf-88ba-32d655e88efd" containerName="payload-pre-processing" containerID="cri-o://3dcbc601edc351292874c97858f175ece0229ad46ca850b54b13bb63cb3bfc6f" gracePeriod=25 Oct 29 23:52:33.754564 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:33.754523 5057 generic.go:356] "Generic (PLEG): container finished" podID="a28590ff-f920-4993-b5f2-d8afb67b3685" containerID="efa1d67631aad66d2812bd3bfb7c2c4f3a989a84b1b89c59701e81985b7cfd04" exitCode=0 Oct 29 23:52:33.755008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:33.754621 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" event={"ID":"a28590ff-f920-4993-b5f2-d8afb67b3685","Type":"ContainerDied","Data":"efa1d67631aad66d2812bd3bfb7c2c4f3a989a84b1b89c59701e81985b7cfd04"} Oct 29 23:52:33.755008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:33.754693 5057 scope.go:122] "RemoveContainer" containerID="3d2dadd12f15bedf8f8b5865e19c4f94f020298f717606fa2fb16a7a4289ce8c" Oct 29 23:52:33.894145 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:33.894116 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:52:33.920642 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:52:33.920604 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:52:34.039065 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.039033 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-podinfo\") pod \"a28590ff-f920-4993-b5f2-d8afb67b3685\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " Oct 29 23:52:34.039295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.039119 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-credential-socket\") pod \"a28590ff-f920-4993-b5f2-d8afb67b3685\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " Oct 29 23:52:34.039295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.039154 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a28590ff-f920-4993-b5f2-d8afb67b3685-kube-api-access-tsp2b\" (UniqueName: \"kubernetes.io/projected/a28590ff-f920-4993-b5f2-d8afb67b3685-kube-api-access-tsp2b\") pod \"a28590ff-f920-4993-b5f2-d8afb67b3685\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " Oct 29 23:52:34.039295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.039178 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a28590ff-f920-4993-b5f2-d8afb67b3685-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28590ff-f920-4993-b5f2-d8afb67b3685-istiod-ca-cert\") pod \"a28590ff-f920-4993-b5f2-d8afb67b3685\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " Oct 29 23:52:34.039295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.039255 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-token\" (UniqueName: \"kubernetes.io/projected/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-token\") pod \"a28590ff-f920-4993-b5f2-d8afb67b3685\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " Oct 29 23:52:34.039295 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.039287 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-workload-socket\") pod \"a28590ff-f920-4993-b5f2-d8afb67b3685\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " Oct 29 23:52:34.039611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.039370 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-envoy\") pod \"a28590ff-f920-4993-b5f2-d8afb67b3685\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " Oct 29 23:52:34.039611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.039419 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-workload-certs\") pod \"a28590ff-f920-4993-b5f2-d8afb67b3685\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " Oct 29 23:52:34.039611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.039519 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-data\") pod \"a28590ff-f920-4993-b5f2-d8afb67b3685\" (UID: \"a28590ff-f920-4993-b5f2-d8afb67b3685\") " Oct 29 23:52:34.040016 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.039493 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-credential-socket" pod "a28590ff-f920-4993-b5f2-d8afb67b3685" (UID: "a28590ff-f920-4993-b5f2-d8afb67b3685"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:34.040016 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.039672 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a28590ff-f920-4993-b5f2-d8afb67b3685-istiod-ca-cert" pod "a28590ff-f920-4993-b5f2-d8afb67b3685" (UID: "a28590ff-f920-4993-b5f2-d8afb67b3685"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:34.040016 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.039944 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-workload-certs" pod "a28590ff-f920-4993-b5f2-d8afb67b3685" (UID: "a28590ff-f920-4993-b5f2-d8afb67b3685"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:34.040016 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.039961 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-data" pod "a28590ff-f920-4993-b5f2-d8afb67b3685" (UID: "a28590ff-f920-4993-b5f2-d8afb67b3685"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:34.040016 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.039980 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-workload-socket" pod "a28590ff-f920-4993-b5f2-d8afb67b3685" (UID: "a28590ff-f920-4993-b5f2-d8afb67b3685"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:34.040453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.040246 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:34.040453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.040264 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:34.040453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.040272 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:34.040453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.040283 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28590ff-f920-4993-b5f2-d8afb67b3685-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:34.040453 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.040296 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:34.041604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.041574 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a28590ff-f920-4993-b5f2-d8afb67b3685-kube-api-access-tsp2b" pod "a28590ff-f920-4993-b5f2-d8afb67b3685" (UID: "a28590ff-f920-4993-b5f2-d8afb67b3685"). InnerVolumeSpecName "kube-api-access-tsp2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:34.041720 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.041706 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-envoy" pod "a28590ff-f920-4993-b5f2-d8afb67b3685" (UID: "a28590ff-f920-4993-b5f2-d8afb67b3685"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:34.042101 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.042076 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-token" pod "a28590ff-f920-4993-b5f2-d8afb67b3685" (UID: "a28590ff-f920-4993-b5f2-d8afb67b3685"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:34.042507 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.042488 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-podinfo" pod "a28590ff-f920-4993-b5f2-d8afb67b3685" (UID: "a28590ff-f920-4993-b5f2-d8afb67b3685"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:52:34.142111 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.142036 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsp2b\" (UniqueName: \"kubernetes.io/projected/a28590ff-f920-4993-b5f2-d8afb67b3685-kube-api-access-tsp2b\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:34.142111 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.142065 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:34.142111 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.142075 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:34.142111 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.142083 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28590ff-f920-4993-b5f2-d8afb67b3685-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:34.764890 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.764857 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" Oct 29 23:52:34.765413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.764857 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr" event={"ID":"a28590ff-f920-4993-b5f2-d8afb67b3685","Type":"ContainerDied","Data":"ddf9b9a8cd7644cd4091d4de49019500c5018571ca1ff7126ccb0eec8861ce92"} Oct 29 23:52:34.765413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.765016 5057 scope.go:122] "RemoveContainer" containerID="efa1d67631aad66d2812bd3bfb7c2c4f3a989a84b1b89c59701e81985b7cfd04" Oct 29 23:52:34.833326 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.833261 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr"] Oct 29 23:52:34.844308 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:34.844275 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-nkdhr"] Oct 29 23:52:35.414325 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:35.414291 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:52:35.433038 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:35.433012 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:52:35.481620 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:35.481584 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a28590ff-f920-4993-b5f2-d8afb67b3685" path="/var/lib/kubelet/pods/a28590ff-f920-4993-b5f2-d8afb67b3685/volumes" Oct 29 23:52:36.177311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:36.177266 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw" podUID="41da29c2-0216-4adf-88ba-32d655e88efd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:52:42.027277 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.027237 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt"] Oct 29 23:52:42.027817 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.027490 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" podUID="4ecbde51-118e-44b3-ae0c-491e4d03f124" containerName="maas-api" containerID="cri-o://227156a9ee0a8dbdcce4e56fd3346310572d83d87e9c252e182e2efb56047a10" gracePeriod=30 Oct 29 23:52:42.366357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.366329 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" Oct 29 23:52:42.455631 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.455579 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ecbde51-118e-44b3-ae0c-491e4d03f124-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ecbde51-118e-44b3-ae0c-491e4d03f124-metrics-certs\") pod \"4ecbde51-118e-44b3-ae0c-491e4d03f124\" (UID: \"4ecbde51-118e-44b3-ae0c-491e4d03f124\") " Oct 29 23:52:42.455831 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.455707 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ecbde51-118e-44b3-ae0c-491e4d03f124-kube-api-access-q6k9p\" (UniqueName: \"kubernetes.io/projected/4ecbde51-118e-44b3-ae0c-491e4d03f124-kube-api-access-q6k9p\") pod \"4ecbde51-118e-44b3-ae0c-491e4d03f124\" (UID: \"4ecbde51-118e-44b3-ae0c-491e4d03f124\") " Oct 29 23:52:42.455831 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.455818 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ecbde51-118e-44b3-ae0c-491e4d03f124-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ecbde51-118e-44b3-ae0c-491e4d03f124-maas-api-tls\") pod \"4ecbde51-118e-44b3-ae0c-491e4d03f124\" (UID: \"4ecbde51-118e-44b3-ae0c-491e4d03f124\") " Oct 29 23:52:42.455977 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.455947 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ecbde51-118e-44b3-ae0c-491e4d03f124-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ecbde51-118e-44b3-ae0c-491e4d03f124-openshift-service-ca\") pod \"4ecbde51-118e-44b3-ae0c-491e4d03f124\" (UID: \"4ecbde51-118e-44b3-ae0c-491e4d03f124\") " Oct 29 23:52:42.456567 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.456534 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ecbde51-118e-44b3-ae0c-491e4d03f124-openshift-service-ca" pod "4ecbde51-118e-44b3-ae0c-491e4d03f124" (UID: "4ecbde51-118e-44b3-ae0c-491e4d03f124"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:42.459227 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.459167 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ecbde51-118e-44b3-ae0c-491e4d03f124-kube-api-access-q6k9p" pod "4ecbde51-118e-44b3-ae0c-491e4d03f124" (UID: "4ecbde51-118e-44b3-ae0c-491e4d03f124"). InnerVolumeSpecName "kube-api-access-q6k9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:42.459379 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.459282 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ecbde51-118e-44b3-ae0c-491e4d03f124-metrics-certs" pod "4ecbde51-118e-44b3-ae0c-491e4d03f124" (UID: "4ecbde51-118e-44b3-ae0c-491e4d03f124"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:52:42.459525 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.459508 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ecbde51-118e-44b3-ae0c-491e4d03f124-maas-api-tls" pod "4ecbde51-118e-44b3-ae0c-491e4d03f124" (UID: "4ecbde51-118e-44b3-ae0c-491e4d03f124"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:52:42.580271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.565446 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ecbde51-118e-44b3-ae0c-491e4d03f124-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:42.581272 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.581241 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ecbde51-118e-44b3-ae0c-491e4d03f124-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:42.581457 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.581442 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ecbde51-118e-44b3-ae0c-491e4d03f124-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:42.581563 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.581549 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6k9p\" (UniqueName: \"kubernetes.io/projected/4ecbde51-118e-44b3-ae0c-491e4d03f124-kube-api-access-q6k9p\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:42.846321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.844261 5057 generic.go:356] "Generic (PLEG): container finished" podID="4ecbde51-118e-44b3-ae0c-491e4d03f124" containerID="227156a9ee0a8dbdcce4e56fd3346310572d83d87e9c252e182e2efb56047a10" exitCode=0 Oct 29 23:52:42.846321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.844382 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" event={"ID":"4ecbde51-118e-44b3-ae0c-491e4d03f124","Type":"ContainerDied","Data":"227156a9ee0a8dbdcce4e56fd3346310572d83d87e9c252e182e2efb56047a10"} Oct 29 23:52:42.846321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.844437 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" event={"ID":"4ecbde51-118e-44b3-ae0c-491e4d03f124","Type":"ContainerDied","Data":"3dae2a198a93af85ecb1545fa838fc5a653683d2acfeef1221b67a82e50b9b03"} Oct 29 23:52:42.846321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.844469 5057 scope.go:122] "RemoveContainer" containerID="227156a9ee0a8dbdcce4e56fd3346310572d83d87e9c252e182e2efb56047a10" Oct 29 23:52:42.846321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.844634 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt" Oct 29 23:52:42.867819 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.867784 5057 scope.go:122] "RemoveContainer" containerID="227156a9ee0a8dbdcce4e56fd3346310572d83d87e9c252e182e2efb56047a10" Oct 29 23:52:42.868181 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:52:42.868152 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"227156a9ee0a8dbdcce4e56fd3346310572d83d87e9c252e182e2efb56047a10\": container with ID starting with 227156a9ee0a8dbdcce4e56fd3346310572d83d87e9c252e182e2efb56047a10 not found: ID does not exist" containerID="227156a9ee0a8dbdcce4e56fd3346310572d83d87e9c252e182e2efb56047a10" Oct 29 23:52:42.868298 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.868206 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"227156a9ee0a8dbdcce4e56fd3346310572d83d87e9c252e182e2efb56047a10"} err="failed to get container status \"227156a9ee0a8dbdcce4e56fd3346310572d83d87e9c252e182e2efb56047a10\": rpc error: code = NotFound desc = could not find container \"227156a9ee0a8dbdcce4e56fd3346310572d83d87e9c252e182e2efb56047a10\": container with ID starting with 227156a9ee0a8dbdcce4e56fd3346310572d83d87e9c252e182e2efb56047a10 not found: ID does not exist" Oct 29 23:52:42.905129 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.905099 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt"] Oct 29 23:52:42.917306 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:42.917277 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-1ee571-649f699b8f-ds8lt"] Oct 29 23:52:43.483166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:43.483133 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ecbde51-118e-44b3-ae0c-491e4d03f124" path="/var/lib/kubelet/pods/4ecbde51-118e-44b3-ae0c-491e4d03f124/volumes" Oct 29 23:52:43.633251 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:43.633198 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59"] Oct 29 23:52:43.633587 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:43.633441 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" podUID="87c126a1-a9e4-4aa9-afe9-78727290d709" containerName="istio-proxy" containerID="cri-o://c535f2396cc502d5ed11aa1ede4ac0a5350729c5e4228b35af44be4f1257e76b" gracePeriod=30 Oct 29 23:52:43.645043 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:43.645012 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf"] Oct 29 23:52:43.645293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:43.645269 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" podUID="abfee661-d5d7-4b6f-8691-aec67412f360" containerName="istio-proxy" containerID="cri-o://fc8617404d84937ea24f36c0f5149954dddcf3ea6f8341b050d1593645a3130f" gracePeriod=30 Oct 29 23:52:46.177073 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.177035 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw" podUID="41da29c2-0216-4adf-88ba-32d655e88efd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:52:46.292611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.292573 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt"] Oct 29 23:52:46.292897 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.292784 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" podUID="52937560-4419-4c5a-b4a1-1918d3963e22" containerName="maas-api" containerID="cri-o://dda7b678beda825f052de432d5493c23f3dcf75ffb7b72732be39a88096021a9" gracePeriod=30 Oct 29 23:52:46.642799 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.642767 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" Oct 29 23:52:46.771079 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.770983 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52937560-4419-4c5a-b4a1-1918d3963e22-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52937560-4419-4c5a-b4a1-1918d3963e22-openshift-service-ca\") pod \"52937560-4419-4c5a-b4a1-1918d3963e22\" (UID: \"52937560-4419-4c5a-b4a1-1918d3963e22\") " Oct 29 23:52:46.771079 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.771049 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52937560-4419-4c5a-b4a1-1918d3963e22-kube-api-access-d64cf\" (UniqueName: \"kubernetes.io/projected/52937560-4419-4c5a-b4a1-1918d3963e22-kube-api-access-d64cf\") pod \"52937560-4419-4c5a-b4a1-1918d3963e22\" (UID: \"52937560-4419-4c5a-b4a1-1918d3963e22\") " Oct 29 23:52:46.771357 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.771327 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52937560-4419-4c5a-b4a1-1918d3963e22-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52937560-4419-4c5a-b4a1-1918d3963e22-metrics-certs\") pod \"52937560-4419-4c5a-b4a1-1918d3963e22\" (UID: \"52937560-4419-4c5a-b4a1-1918d3963e22\") " Oct 29 23:52:46.771427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.771366 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52937560-4419-4c5a-b4a1-1918d3963e22-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52937560-4419-4c5a-b4a1-1918d3963e22-maas-api-tls\") pod \"52937560-4419-4c5a-b4a1-1918d3963e22\" (UID: \"52937560-4419-4c5a-b4a1-1918d3963e22\") " Oct 29 23:52:46.771427 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.771368 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52937560-4419-4c5a-b4a1-1918d3963e22-openshift-service-ca" pod "52937560-4419-4c5a-b4a1-1918d3963e22" (UID: "52937560-4419-4c5a-b4a1-1918d3963e22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:46.771950 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.771928 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52937560-4419-4c5a-b4a1-1918d3963e22-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:46.773338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.773313 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52937560-4419-4c5a-b4a1-1918d3963e22-kube-api-access-d64cf" pod "52937560-4419-4c5a-b4a1-1918d3963e22" (UID: "52937560-4419-4c5a-b4a1-1918d3963e22"). InnerVolumeSpecName "kube-api-access-d64cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:46.773779 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.773756 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52937560-4419-4c5a-b4a1-1918d3963e22-maas-api-tls" pod "52937560-4419-4c5a-b4a1-1918d3963e22" (UID: "52937560-4419-4c5a-b4a1-1918d3963e22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:52:46.773870 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.773785 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52937560-4419-4c5a-b4a1-1918d3963e22-metrics-certs" pod "52937560-4419-4c5a-b4a1-1918d3963e22" (UID: "52937560-4419-4c5a-b4a1-1918d3963e22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:52:46.874224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.874173 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52937560-4419-4c5a-b4a1-1918d3963e22-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:46.874224 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.874225 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52937560-4419-4c5a-b4a1-1918d3963e22-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:46.874458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.874241 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d64cf\" (UniqueName: \"kubernetes.io/projected/52937560-4419-4c5a-b4a1-1918d3963e22-kube-api-access-d64cf\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:46.880779 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.880747 5057 generic.go:356] "Generic (PLEG): container finished" podID="52937560-4419-4c5a-b4a1-1918d3963e22" containerID="dda7b678beda825f052de432d5493c23f3dcf75ffb7b72732be39a88096021a9" exitCode=0 Oct 29 23:52:46.880928 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.880792 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" Oct 29 23:52:46.880928 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.880796 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" event={"ID":"52937560-4419-4c5a-b4a1-1918d3963e22","Type":"ContainerDied","Data":"dda7b678beda825f052de432d5493c23f3dcf75ffb7b72732be39a88096021a9"} Oct 29 23:52:46.880928 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.880913 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt" event={"ID":"52937560-4419-4c5a-b4a1-1918d3963e22","Type":"ContainerDied","Data":"0831a17c0927e90a1a30e3993f88efb816030e4a35976ed88a8d8f22fc5ce0da"} Oct 29 23:52:46.881052 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.880937 5057 scope.go:122] "RemoveContainer" containerID="dda7b678beda825f052de432d5493c23f3dcf75ffb7b72732be39a88096021a9" Oct 29 23:52:46.891592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.891572 5057 scope.go:122] "RemoveContainer" containerID="dda7b678beda825f052de432d5493c23f3dcf75ffb7b72732be39a88096021a9" Oct 29 23:52:46.891938 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:52:46.891907 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dda7b678beda825f052de432d5493c23f3dcf75ffb7b72732be39a88096021a9\": container with ID starting with dda7b678beda825f052de432d5493c23f3dcf75ffb7b72732be39a88096021a9 not found: ID does not exist" containerID="dda7b678beda825f052de432d5493c23f3dcf75ffb7b72732be39a88096021a9" Oct 29 23:52:46.892040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.891944 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda7b678beda825f052de432d5493c23f3dcf75ffb7b72732be39a88096021a9"} err="failed to get container status \"dda7b678beda825f052de432d5493c23f3dcf75ffb7b72732be39a88096021a9\": rpc error: code = NotFound desc = could not find container \"dda7b678beda825f052de432d5493c23f3dcf75ffb7b72732be39a88096021a9\": container with ID starting with dda7b678beda825f052de432d5493c23f3dcf75ffb7b72732be39a88096021a9 not found: ID does not exist" Oct 29 23:52:46.932535 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.932502 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt"] Oct 29 23:52:46.936398 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:46.936373 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-adc7d7-7f955498c5-d5grt"] Oct 29 23:52:47.482095 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:47.482062 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52937560-4419-4c5a-b4a1-1918d3963e22" path="/var/lib/kubelet/pods/52937560-4419-4c5a-b4a1-1918d3963e22/volumes" Oct 29 23:52:48.899473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:48.899437 5057 generic.go:356] "Generic (PLEG): container finished" podID="abfee661-d5d7-4b6f-8691-aec67412f360" containerID="fc8617404d84937ea24f36c0f5149954dddcf3ea6f8341b050d1593645a3130f" exitCode=0 Oct 29 23:52:48.900010 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:48.899562 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" event={"ID":"abfee661-d5d7-4b6f-8691-aec67412f360","Type":"ContainerDied","Data":"fc8617404d84937ea24f36c0f5149954dddcf3ea6f8341b050d1593645a3130f"} Oct 29 23:52:48.902299 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:48.902232 5057 generic.go:356] "Generic (PLEG): container finished" podID="87c126a1-a9e4-4aa9-afe9-78727290d709" containerID="c535f2396cc502d5ed11aa1ede4ac0a5350729c5e4228b35af44be4f1257e76b" exitCode=0 Oct 29 23:52:48.902423 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:48.902316 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" event={"ID":"87c126a1-a9e4-4aa9-afe9-78727290d709","Type":"ContainerDied","Data":"c535f2396cc502d5ed11aa1ede4ac0a5350729c5e4228b35af44be4f1257e76b"} Oct 29 23:52:48.936745 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:48.936718 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:52:48.966525 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:48.966499 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:52:49.127007 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.126961 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-workload-certs\") pod \"87c126a1-a9e4-4aa9-afe9-78727290d709\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " Oct 29 23:52:49.154147 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.147311 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-credential-socket\") pod \"abfee661-d5d7-4b6f-8691-aec67412f360\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " Oct 29 23:52:49.168974 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.134326 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-workload-certs" pod "87c126a1-a9e4-4aa9-afe9-78727290d709" (UID: "87c126a1-a9e4-4aa9-afe9-78727290d709"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.169140 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.148168 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-credential-socket" pod "abfee661-d5d7-4b6f-8691-aec67412f360" (UID: "abfee661-d5d7-4b6f-8691-aec67412f360"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.169140 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.169019 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-workload-socket\") pod \"87c126a1-a9e4-4aa9-afe9-78727290d709\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " Oct 29 23:52:49.169140 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.169064 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-workload-socket\") pod \"abfee661-d5d7-4b6f-8691-aec67412f360\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " Oct 29 23:52:49.169140 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.169128 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-token\" (UniqueName: \"kubernetes.io/projected/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-token\") pod \"87c126a1-a9e4-4aa9-afe9-78727290d709\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " Oct 29 23:52:49.169382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.169200 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-data\") pod \"87c126a1-a9e4-4aa9-afe9-78727290d709\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " Oct 29 23:52:49.169382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.169230 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abfee661-d5d7-4b6f-8691-aec67412f360-kube-api-access-kss52\" (UniqueName: \"kubernetes.io/projected/abfee661-d5d7-4b6f-8691-aec67412f360-kube-api-access-kss52\") pod \"abfee661-d5d7-4b6f-8691-aec67412f360\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " Oct 29 23:52:49.169382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.169266 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-istio-envoy\") pod \"abfee661-d5d7-4b6f-8691-aec67412f360\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " Oct 29 23:52:49.169523 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.169433 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87c126a1-a9e4-4aa9-afe9-78727290d709-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c126a1-a9e4-4aa9-afe9-78727290d709-istiod-ca-cert\") pod \"87c126a1-a9e4-4aa9-afe9-78727290d709\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " Oct 29 23:52:49.169523 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.169460 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abfee661-d5d7-4b6f-8691-aec67412f360-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfee661-d5d7-4b6f-8691-aec67412f360-istiod-ca-cert\") pod \"abfee661-d5d7-4b6f-8691-aec67412f360\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " Oct 29 23:52:49.169523 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.169494 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abfee661-d5d7-4b6f-8691-aec67412f360-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfee661-d5d7-4b6f-8691-aec67412f360-istio-podinfo\") pod \"abfee661-d5d7-4b6f-8691-aec67412f360\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " Oct 29 23:52:49.169662 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.169532 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-credential-socket\") pod \"87c126a1-a9e4-4aa9-afe9-78727290d709\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " Oct 29 23:52:49.169662 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.169570 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-istio-data\") pod \"abfee661-d5d7-4b6f-8691-aec67412f360\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " Oct 29 23:52:49.169662 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.169614 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-envoy\") pod \"87c126a1-a9e4-4aa9-afe9-78727290d709\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " Oct 29 23:52:49.169809 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.169671 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abfee661-d5d7-4b6f-8691-aec67412f360-istio-token\" (UniqueName: \"kubernetes.io/projected/abfee661-d5d7-4b6f-8691-aec67412f360-istio-token\") pod \"abfee661-d5d7-4b6f-8691-aec67412f360\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " Oct 29 23:52:49.169809 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.169723 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-workload-certs\") pod \"abfee661-d5d7-4b6f-8691-aec67412f360\" (UID: \"abfee661-d5d7-4b6f-8691-aec67412f360\") " Oct 29 23:52:49.169809 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.169771 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c126a1-a9e4-4aa9-afe9-78727290d709-kube-api-access-mgltd\" (UniqueName: \"kubernetes.io/projected/87c126a1-a9e4-4aa9-afe9-78727290d709-kube-api-access-mgltd\") pod \"87c126a1-a9e4-4aa9-afe9-78727290d709\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " Oct 29 23:52:49.169952 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.169824 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-podinfo\") pod \"87c126a1-a9e4-4aa9-afe9-78727290d709\" (UID: \"87c126a1-a9e4-4aa9-afe9-78727290d709\") " Oct 29 23:52:49.170462 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.170438 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.170462 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.170462 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.171347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.171202 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abfee661-d5d7-4b6f-8691-aec67412f360-istiod-ca-cert" pod "abfee661-d5d7-4b6f-8691-aec67412f360" (UID: "abfee661-d5d7-4b6f-8691-aec67412f360"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:49.172127 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.171482 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-credential-socket" pod "87c126a1-a9e4-4aa9-afe9-78727290d709" (UID: "87c126a1-a9e4-4aa9-afe9-78727290d709"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.172127 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.171802 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-workload-socket" pod "87c126a1-a9e4-4aa9-afe9-78727290d709" (UID: "87c126a1-a9e4-4aa9-afe9-78727290d709"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.172127 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.171864 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-workload-certs" pod "abfee661-d5d7-4b6f-8691-aec67412f360" (UID: "abfee661-d5d7-4b6f-8691-aec67412f360"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.172724 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.172377 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-workload-socket" pod "abfee661-d5d7-4b6f-8691-aec67412f360" (UID: "abfee661-d5d7-4b6f-8691-aec67412f360"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.172724 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.172610 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-istio-data" pod "abfee661-d5d7-4b6f-8691-aec67412f360" (UID: "abfee661-d5d7-4b6f-8691-aec67412f360"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.172908 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.172869 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-data" pod "87c126a1-a9e4-4aa9-afe9-78727290d709" (UID: "87c126a1-a9e4-4aa9-afe9-78727290d709"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.173170 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.173141 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87c126a1-a9e4-4aa9-afe9-78727290d709-istiod-ca-cert" pod "87c126a1-a9e4-4aa9-afe9-78727290d709" (UID: "87c126a1-a9e4-4aa9-afe9-78727290d709"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:49.176608 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.176583 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-podinfo" pod "87c126a1-a9e4-4aa9-afe9-78727290d709" (UID: "87c126a1-a9e4-4aa9-afe9-78727290d709"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:52:49.178367 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.178343 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-envoy" pod "87c126a1-a9e4-4aa9-afe9-78727290d709" (UID: "87c126a1-a9e4-4aa9-afe9-78727290d709"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.178457 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.178362 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abfee661-d5d7-4b6f-8691-aec67412f360-istio-podinfo" pod "abfee661-d5d7-4b6f-8691-aec67412f360" (UID: "abfee661-d5d7-4b6f-8691-aec67412f360"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:52:49.178457 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.178369 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c126a1-a9e4-4aa9-afe9-78727290d709-kube-api-access-mgltd" pod "87c126a1-a9e4-4aa9-afe9-78727290d709" (UID: "87c126a1-a9e4-4aa9-afe9-78727290d709"). InnerVolumeSpecName "kube-api-access-mgltd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:49.178457 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.178388 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfee661-d5d7-4b6f-8691-aec67412f360-istio-token" pod "abfee661-d5d7-4b6f-8691-aec67412f360" (UID: "abfee661-d5d7-4b6f-8691-aec67412f360"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:49.178457 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.178369 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-istio-envoy" pod "abfee661-d5d7-4b6f-8691-aec67412f360" (UID: "abfee661-d5d7-4b6f-8691-aec67412f360"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.178608 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.178510 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfee661-d5d7-4b6f-8691-aec67412f360-kube-api-access-kss52" pod "abfee661-d5d7-4b6f-8691-aec67412f360" (UID: "abfee661-d5d7-4b6f-8691-aec67412f360"). InnerVolumeSpecName "kube-api-access-kss52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:49.180240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.180220 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-token" pod "87c126a1-a9e4-4aa9-afe9-78727290d709" (UID: "87c126a1-a9e4-4aa9-afe9-78727290d709"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:49.271580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.271543 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c126a1-a9e4-4aa9-afe9-78727290d709-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.271580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.271578 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfee661-d5d7-4b6f-8691-aec67412f360-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.271580 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.271586 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfee661-d5d7-4b6f-8691-aec67412f360-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.271827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.271595 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.271827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.271604 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.271827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.271612 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.271827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.271620 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfee661-d5d7-4b6f-8691-aec67412f360-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.271827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.271627 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.271827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.271635 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgltd\" (UniqueName: \"kubernetes.io/projected/87c126a1-a9e4-4aa9-afe9-78727290d709-kube-api-access-mgltd\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.271827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.271644 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.271827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.271653 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.271827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.271661 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.271827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.271669 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.271827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.271676 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c126a1-a9e4-4aa9-afe9-78727290d709-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.271827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.271684 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kss52\" (UniqueName: \"kubernetes.io/projected/abfee661-d5d7-4b6f-8691-aec67412f360-kube-api-access-kss52\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.271827 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.271691 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfee661-d5d7-4b6f-8691-aec67412f360-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.910218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.910117 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" event={"ID":"abfee661-d5d7-4b6f-8691-aec67412f360","Type":"ContainerDied","Data":"09145a9b9f9fc33ef4bcaf6e27d17f3fd30e7bafaa7a539c8822bd3abe29c2b7"} Oct 29 23:52:49.910218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.910123 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf" Oct 29 23:52:49.910772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.910259 5057 scope.go:122] "RemoveContainer" containerID="fc8617404d84937ea24f36c0f5149954dddcf3ea6f8341b050d1593645a3130f" Oct 29 23:52:49.911740 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.911711 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" event={"ID":"87c126a1-a9e4-4aa9-afe9-78727290d709","Type":"ContainerDied","Data":"61fe47447b0a58d7a7c609e11c4ce843fa164bbf5af4936accda06b230eb9747"} Oct 29 23:52:49.911885 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.911722 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59" Oct 29 23:52:49.927460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.924645 5057 scope.go:122] "RemoveContainer" containerID="c535f2396cc502d5ed11aa1ede4ac0a5350729c5e4228b35af44be4f1257e76b" Oct 29 23:52:49.976903 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.976865 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf"] Oct 29 23:52:49.983643 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:49.983614 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-d6zxf"] Oct 29 23:52:50.017026 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:50.016988 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59"] Oct 29 23:52:50.022727 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:50.022699 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-s2j59"] Oct 29 23:52:51.367020 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:51.366987 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw" Oct 29 23:52:51.407069 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:51.407037 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41da29c2-0216-4adf-88ba-32d655e88efd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41da29c2-0216-4adf-88ba-32d655e88efd-plugins-config-volume\") pod \"41da29c2-0216-4adf-88ba-32d655e88efd\" (UID: \"41da29c2-0216-4adf-88ba-32d655e88efd\") " Oct 29 23:52:51.407343 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:51.407199 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41da29c2-0216-4adf-88ba-32d655e88efd-kube-api-access-8785n\" (UniqueName: \"kubernetes.io/projected/41da29c2-0216-4adf-88ba-32d655e88efd-kube-api-access-8785n\") pod \"41da29c2-0216-4adf-88ba-32d655e88efd\" (UID: \"41da29c2-0216-4adf-88ba-32d655e88efd\") " Oct 29 23:52:51.407794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:51.407481 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41da29c2-0216-4adf-88ba-32d655e88efd-plugins-config-volume" pod "41da29c2-0216-4adf-88ba-32d655e88efd" (UID: "41da29c2-0216-4adf-88ba-32d655e88efd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:51.407794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:51.407635 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41da29c2-0216-4adf-88ba-32d655e88efd-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:51.410373 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:51.410341 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41da29c2-0216-4adf-88ba-32d655e88efd-kube-api-access-8785n" pod "41da29c2-0216-4adf-88ba-32d655e88efd" (UID: "41da29c2-0216-4adf-88ba-32d655e88efd"). InnerVolumeSpecName "kube-api-access-8785n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:51.483369 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:51.483325 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87c126a1-a9e4-4aa9-afe9-78727290d709" path="/var/lib/kubelet/pods/87c126a1-a9e4-4aa9-afe9-78727290d709/volumes" Oct 29 23:52:51.483927 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:51.483904 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abfee661-d5d7-4b6f-8691-aec67412f360" path="/var/lib/kubelet/pods/abfee661-d5d7-4b6f-8691-aec67412f360/volumes" Oct 29 23:52:51.508700 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:51.508675 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8785n\" (UniqueName: \"kubernetes.io/projected/41da29c2-0216-4adf-88ba-32d655e88efd-kube-api-access-8785n\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:52:51.928721 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:51.928680 5057 generic.go:356] "Generic (PLEG): container finished" podID="41da29c2-0216-4adf-88ba-32d655e88efd" containerID="3dcbc601edc351292874c97858f175ece0229ad46ca850b54b13bb63cb3bfc6f" exitCode=0 Oct 29 23:52:51.928721 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:51.928732 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw" Oct 29 23:52:51.929018 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:51.928786 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw" event={"ID":"41da29c2-0216-4adf-88ba-32d655e88efd","Type":"ContainerDied","Data":"3dcbc601edc351292874c97858f175ece0229ad46ca850b54b13bb63cb3bfc6f"} Oct 29 23:52:51.929018 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:51.928864 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw" event={"ID":"41da29c2-0216-4adf-88ba-32d655e88efd","Type":"ContainerDied","Data":"8726510575d9a0134d5748d34456f48e6e01785e5f5156001fb2b1ab6233d8e7"} Oct 29 23:52:51.929018 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:51.928896 5057 scope.go:122] "RemoveContainer" containerID="3dcbc601edc351292874c97858f175ece0229ad46ca850b54b13bb63cb3bfc6f" Oct 29 23:52:51.940090 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:51.940063 5057 scope.go:122] "RemoveContainer" containerID="3dcbc601edc351292874c97858f175ece0229ad46ca850b54b13bb63cb3bfc6f" Oct 29 23:52:51.940484 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:52:51.940453 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dcbc601edc351292874c97858f175ece0229ad46ca850b54b13bb63cb3bfc6f\": container with ID starting with 3dcbc601edc351292874c97858f175ece0229ad46ca850b54b13bb63cb3bfc6f not found: ID does not exist" containerID="3dcbc601edc351292874c97858f175ece0229ad46ca850b54b13bb63cb3bfc6f" Oct 29 23:52:51.940574 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:51.940496 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dcbc601edc351292874c97858f175ece0229ad46ca850b54b13bb63cb3bfc6f"} err="failed to get container status \"3dcbc601edc351292874c97858f175ece0229ad46ca850b54b13bb63cb3bfc6f\": rpc error: code = NotFound desc = could not find container \"3dcbc601edc351292874c97858f175ece0229ad46ca850b54b13bb63cb3bfc6f\": container with ID starting with 3dcbc601edc351292874c97858f175ece0229ad46ca850b54b13bb63cb3bfc6f not found: ID does not exist" Oct 29 23:52:51.981654 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:51.981622 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw"] Oct 29 23:52:51.986094 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:51.986067 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-206095-67446cdc55-qbhtw"] Oct 29 23:52:53.481675 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:53.481630 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41da29c2-0216-4adf-88ba-32d655e88efd" path="/var/lib/kubelet/pods/41da29c2-0216-4adf-88ba-32d655e88efd/volumes" Oct 29 23:52:53.828273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:52:53.821860 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:53:03.968563 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:53:03.968517 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:53:09.832211 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:09.819132 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:53:15.746079 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:15.746041 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj"] Oct 29 23:53:15.777964 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:15.777931 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9"] Oct 29 23:53:20.746565 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:20.746512 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj" podUID="153fed7b-8844-4d52-a5ad-0b6d8eda52e7" containerName="payload-processing" containerID="cri-o://ced5fbebedd978b16c23844f46cf1105fd2c4249a979349af808685c0e2c6b76" gracePeriod=25 Oct 29 23:53:20.778484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:20.778442 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9" podUID="5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b" containerName="payload-pre-processing" containerID="cri-o://9c8f22e461d7d5785f7faa0e92c98a58940a94c46624e54bc047b508bb089055" gracePeriod=25 Oct 29 23:53:21.278883 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:21.278848 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4"] Oct 29 23:53:21.279252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:21.279146 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" podUID="c6508b22-be11-4d60-80bd-42a88a299853" containerName="istio-proxy" containerID="cri-o://b53553346862a1e1bc5277b978ef73b670bc1ae755b1ff086bbf45f663e785c5" gracePeriod=30 Oct 29 23:53:21.767457 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:21.767413 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9" podUID="5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:53:22.068032 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:22.067955 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:53:22.068032 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:22.068020 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" podUID="c6508b22-be11-4d60-80bd-42a88a299853" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:53:22.980115 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:22.980081 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb"] Oct 29 23:53:22.980597 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:22.980343 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" podUID="4104e017-b65b-404f-8226-71df8ca5c903" containerName="istio-proxy" containerID="cri-o://59eba49db777c8dfcd6a586e44e7b15743786122ad608008d103438840d62135" gracePeriod=30 Oct 29 23:53:23.789261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:23.789218 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj" podUID="153fed7b-8844-4d52-a5ad-0b6d8eda52e7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:53:23.831443 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:23.831402 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:53:26.554507 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.554482 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:53:26.648492 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.648453 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6508b22-be11-4d60-80bd-42a88a299853-kube-api-access-f946r\" (UniqueName: \"kubernetes.io/projected/c6508b22-be11-4d60-80bd-42a88a299853-kube-api-access-f946r\") pod \"c6508b22-be11-4d60-80bd-42a88a299853\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " Oct 29 23:53:26.648810 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.648518 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-istio-envoy\") pod \"c6508b22-be11-4d60-80bd-42a88a299853\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " Oct 29 23:53:26.648810 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.648566 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6508b22-be11-4d60-80bd-42a88a299853-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6508b22-be11-4d60-80bd-42a88a299853-istiod-ca-cert\") pod \"c6508b22-be11-4d60-80bd-42a88a299853\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " Oct 29 23:53:26.648810 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.648605 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6508b22-be11-4d60-80bd-42a88a299853-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6508b22-be11-4d60-80bd-42a88a299853-istio-podinfo\") pod \"c6508b22-be11-4d60-80bd-42a88a299853\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " Oct 29 23:53:26.648810 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.648708 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-istio-data\") pod \"c6508b22-be11-4d60-80bd-42a88a299853\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " Oct 29 23:53:26.648810 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.648749 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-workload-certs\") pod \"c6508b22-be11-4d60-80bd-42a88a299853\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " Oct 29 23:53:26.649124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.648811 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6508b22-be11-4d60-80bd-42a88a299853-istio-token\" (UniqueName: \"kubernetes.io/projected/c6508b22-be11-4d60-80bd-42a88a299853-istio-token\") pod \"c6508b22-be11-4d60-80bd-42a88a299853\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " Oct 29 23:53:26.649124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.648843 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-workload-socket\") pod \"c6508b22-be11-4d60-80bd-42a88a299853\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " Oct 29 23:53:26.649124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.648875 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-credential-socket\") pod \"c6508b22-be11-4d60-80bd-42a88a299853\" (UID: \"c6508b22-be11-4d60-80bd-42a88a299853\") " Oct 29 23:53:26.649335 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.649115 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-workload-certs" pod "c6508b22-be11-4d60-80bd-42a88a299853" (UID: "c6508b22-be11-4d60-80bd-42a88a299853"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:26.649403 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.649369 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-workload-socket" pod "c6508b22-be11-4d60-80bd-42a88a299853" (UID: "c6508b22-be11-4d60-80bd-42a88a299853"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:26.649403 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.649384 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-istio-data" pod "c6508b22-be11-4d60-80bd-42a88a299853" (UID: "c6508b22-be11-4d60-80bd-42a88a299853"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:26.649655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.649613 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:26.649655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.649639 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:26.649655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.649639 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6508b22-be11-4d60-80bd-42a88a299853-istiod-ca-cert" pod "c6508b22-be11-4d60-80bd-42a88a299853" (UID: "c6508b22-be11-4d60-80bd-42a88a299853"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:53:26.649655 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.649652 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:26.649956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.649695 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-credential-socket" pod "c6508b22-be11-4d60-80bd-42a88a299853" (UID: "c6508b22-be11-4d60-80bd-42a88a299853"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:26.651293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.651253 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6508b22-be11-4d60-80bd-42a88a299853-kube-api-access-f946r" pod "c6508b22-be11-4d60-80bd-42a88a299853" (UID: "c6508b22-be11-4d60-80bd-42a88a299853"). InnerVolumeSpecName "kube-api-access-f946r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:26.652083 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.652059 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6508b22-be11-4d60-80bd-42a88a299853-istio-token" pod "c6508b22-be11-4d60-80bd-42a88a299853" (UID: "c6508b22-be11-4d60-80bd-42a88a299853"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:26.652301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.652121 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6508b22-be11-4d60-80bd-42a88a299853-istio-podinfo" pod "c6508b22-be11-4d60-80bd-42a88a299853" (UID: "c6508b22-be11-4d60-80bd-42a88a299853"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:53:26.652560 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.652541 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-istio-envoy" pod "c6508b22-be11-4d60-80bd-42a88a299853" (UID: "c6508b22-be11-4d60-80bd-42a88a299853"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:26.750688 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.750652 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6508b22-be11-4d60-80bd-42a88a299853-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:26.750688 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.750683 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:26.750907 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.750696 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f946r\" (UniqueName: \"kubernetes.io/projected/c6508b22-be11-4d60-80bd-42a88a299853-kube-api-access-f946r\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:26.750907 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.750711 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6508b22-be11-4d60-80bd-42a88a299853-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:26.750907 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.750722 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6508b22-be11-4d60-80bd-42a88a299853-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:26.750907 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:26.750735 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6508b22-be11-4d60-80bd-42a88a299853-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:27.206996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:27.206962 5057 generic.go:356] "Generic (PLEG): container finished" podID="c6508b22-be11-4d60-80bd-42a88a299853" containerID="b53553346862a1e1bc5277b978ef73b670bc1ae755b1ff086bbf45f663e785c5" exitCode=0 Oct 29 23:53:27.206996 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:27.206999 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" event={"ID":"c6508b22-be11-4d60-80bd-42a88a299853","Type":"ContainerDied","Data":"b53553346862a1e1bc5277b978ef73b670bc1ae755b1ff086bbf45f663e785c5"} Oct 29 23:53:27.207322 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:27.207015 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" Oct 29 23:53:27.207322 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:27.207038 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4" event={"ID":"c6508b22-be11-4d60-80bd-42a88a299853","Type":"ContainerDied","Data":"a2fe8d8718ea6dcbc49659a1017125876c3581e8f3f1c8dfa5909b3c6cef2769"} Oct 29 23:53:27.207322 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:27.207061 5057 scope.go:122] "RemoveContainer" containerID="b53553346862a1e1bc5277b978ef73b670bc1ae755b1ff086bbf45f663e785c5" Oct 29 23:53:27.218804 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:27.218788 5057 scope.go:122] "RemoveContainer" containerID="2d557413580e7e4deb2404281cec8352a50820c901754479b16340047f6134fc" Oct 29 23:53:27.228226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:27.228181 5057 scope.go:122] "RemoveContainer" containerID="b53553346862a1e1bc5277b978ef73b670bc1ae755b1ff086bbf45f663e785c5" Oct 29 23:53:27.228521 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:53:27.228495 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b53553346862a1e1bc5277b978ef73b670bc1ae755b1ff086bbf45f663e785c5\": container with ID starting with b53553346862a1e1bc5277b978ef73b670bc1ae755b1ff086bbf45f663e785c5 not found: ID does not exist" containerID="b53553346862a1e1bc5277b978ef73b670bc1ae755b1ff086bbf45f663e785c5" Oct 29 23:53:27.228585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:27.228533 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b53553346862a1e1bc5277b978ef73b670bc1ae755b1ff086bbf45f663e785c5"} err="failed to get container status \"b53553346862a1e1bc5277b978ef73b670bc1ae755b1ff086bbf45f663e785c5\": rpc error: code = NotFound desc = could not find container \"b53553346862a1e1bc5277b978ef73b670bc1ae755b1ff086bbf45f663e785c5\": container with ID starting with b53553346862a1e1bc5277b978ef73b670bc1ae755b1ff086bbf45f663e785c5 not found: ID does not exist" Oct 29 23:53:27.228585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:27.228569 5057 scope.go:122] "RemoveContainer" containerID="2d557413580e7e4deb2404281cec8352a50820c901754479b16340047f6134fc" Oct 29 23:53:27.228868 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:53:27.228849 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d557413580e7e4deb2404281cec8352a50820c901754479b16340047f6134fc\": container with ID starting with 2d557413580e7e4deb2404281cec8352a50820c901754479b16340047f6134fc not found: ID does not exist" containerID="2d557413580e7e4deb2404281cec8352a50820c901754479b16340047f6134fc" Oct 29 23:53:27.228914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:27.228875 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d557413580e7e4deb2404281cec8352a50820c901754479b16340047f6134fc"} err="failed to get container status \"2d557413580e7e4deb2404281cec8352a50820c901754479b16340047f6134fc\": rpc error: code = NotFound desc = could not find container \"2d557413580e7e4deb2404281cec8352a50820c901754479b16340047f6134fc\": container with ID starting with 2d557413580e7e4deb2404281cec8352a50820c901754479b16340047f6134fc not found: ID does not exist" Oct 29 23:53:27.271489 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:27.260775 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4"] Oct 29 23:53:27.275033 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:27.275006 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-45036b-openshift-default-5cd77fcd4f-vdmj4"] Oct 29 23:53:27.481729 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:27.481643 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6508b22-be11-4d60-80bd-42a88a299853" path="/var/lib/kubelet/pods/c6508b22-be11-4d60-80bd-42a88a299853/volumes" Oct 29 23:53:28.215143 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.215108 5057 generic.go:356] "Generic (PLEG): container finished" podID="4104e017-b65b-404f-8226-71df8ca5c903" containerID="59eba49db777c8dfcd6a586e44e7b15743786122ad608008d103438840d62135" exitCode=0 Oct 29 23:53:28.215595 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.215179 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" event={"ID":"4104e017-b65b-404f-8226-71df8ca5c903","Type":"ContainerDied","Data":"59eba49db777c8dfcd6a586e44e7b15743786122ad608008d103438840d62135"} Oct 29 23:53:28.215595 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.215254 5057 scope.go:122] "RemoveContainer" containerID="7f930136d37f6277742099dba83ed718bf209a41345210ea80deac27433f6d9e" Oct 29 23:53:28.661814 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.661789 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:53:28.774625 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.774542 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4104e017-b65b-404f-8226-71df8ca5c903-kube-api-access-kl5gz\" (UniqueName: \"kubernetes.io/projected/4104e017-b65b-404f-8226-71df8ca5c903-kube-api-access-kl5gz\") pod \"4104e017-b65b-404f-8226-71df8ca5c903\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " Oct 29 23:53:28.774780 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.774621 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4104e017-b65b-404f-8226-71df8ca5c903-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4104e017-b65b-404f-8226-71df8ca5c903-istiod-ca-cert\") pod \"4104e017-b65b-404f-8226-71df8ca5c903\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " Oct 29 23:53:28.774780 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.774659 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-istio-envoy\") pod \"4104e017-b65b-404f-8226-71df8ca5c903\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " Oct 29 23:53:28.774904 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.774810 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-workload-socket\") pod \"4104e017-b65b-404f-8226-71df8ca5c903\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " Oct 29 23:53:28.774965 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.774917 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4104e017-b65b-404f-8226-71df8ca5c903-istio-token\" (UniqueName: \"kubernetes.io/projected/4104e017-b65b-404f-8226-71df8ca5c903-istio-token\") pod \"4104e017-b65b-404f-8226-71df8ca5c903\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " Oct 29 23:53:28.775035 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.775020 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-istio-data\") pod \"4104e017-b65b-404f-8226-71df8ca5c903\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " Oct 29 23:53:28.775128 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.775035 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-workload-socket" pod "4104e017-b65b-404f-8226-71df8ca5c903" (UID: "4104e017-b65b-404f-8226-71df8ca5c903"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:28.775128 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.775060 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4104e017-b65b-404f-8226-71df8ca5c903-istiod-ca-cert" pod "4104e017-b65b-404f-8226-71df8ca5c903" (UID: "4104e017-b65b-404f-8226-71df8ca5c903"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:53:28.775128 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.775084 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-workload-certs\") pod \"4104e017-b65b-404f-8226-71df8ca5c903\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " Oct 29 23:53:28.775128 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.775123 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4104e017-b65b-404f-8226-71df8ca5c903-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4104e017-b65b-404f-8226-71df8ca5c903-istio-podinfo\") pod \"4104e017-b65b-404f-8226-71df8ca5c903\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " Oct 29 23:53:28.775382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.775149 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-credential-socket\") pod \"4104e017-b65b-404f-8226-71df8ca5c903\" (UID: \"4104e017-b65b-404f-8226-71df8ca5c903\") " Oct 29 23:53:28.775382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.775316 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-istio-data" pod "4104e017-b65b-404f-8226-71df8ca5c903" (UID: "4104e017-b65b-404f-8226-71df8ca5c903"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:28.775496 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.775394 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-workload-certs" pod "4104e017-b65b-404f-8226-71df8ca5c903" (UID: "4104e017-b65b-404f-8226-71df8ca5c903"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:28.775635 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.775612 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-credential-socket" pod "4104e017-b65b-404f-8226-71df8ca5c903" (UID: "4104e017-b65b-404f-8226-71df8ca5c903"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:28.775873 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.775854 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:28.775967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.775876 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:28.775967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.775890 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:28.775967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.775904 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:28.775967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.775917 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4104e017-b65b-404f-8226-71df8ca5c903-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:28.777282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.777259 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-istio-envoy" pod "4104e017-b65b-404f-8226-71df8ca5c903" (UID: "4104e017-b65b-404f-8226-71df8ca5c903"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:28.777396 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.777381 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4104e017-b65b-404f-8226-71df8ca5c903-istio-podinfo" pod "4104e017-b65b-404f-8226-71df8ca5c903" (UID: "4104e017-b65b-404f-8226-71df8ca5c903"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:53:28.777496 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.777483 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4104e017-b65b-404f-8226-71df8ca5c903-kube-api-access-kl5gz" pod "4104e017-b65b-404f-8226-71df8ca5c903" (UID: "4104e017-b65b-404f-8226-71df8ca5c903"). InnerVolumeSpecName "kube-api-access-kl5gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:28.777535 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.777503 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4104e017-b65b-404f-8226-71df8ca5c903-istio-token" pod "4104e017-b65b-404f-8226-71df8ca5c903" (UID: "4104e017-b65b-404f-8226-71df8ca5c903"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:28.877112 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.877081 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl5gz\" (UniqueName: \"kubernetes.io/projected/4104e017-b65b-404f-8226-71df8ca5c903-kube-api-access-kl5gz\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:28.877112 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.877109 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4104e017-b65b-404f-8226-71df8ca5c903-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:28.877112 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.877120 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4104e017-b65b-404f-8226-71df8ca5c903-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:28.877403 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:28.877130 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4104e017-b65b-404f-8226-71df8ca5c903-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:29.223829 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:29.223792 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" event={"ID":"4104e017-b65b-404f-8226-71df8ca5c903","Type":"ContainerDied","Data":"f877e453847c930e52fb84d46364106a8f3ffc07e90aa952f47bc9e3bf5ffb90"} Oct 29 23:53:29.224330 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:29.223857 5057 scope.go:122] "RemoveContainer" containerID="59eba49db777c8dfcd6a586e44e7b15743786122ad608008d103438840d62135" Oct 29 23:53:29.224330 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:29.223805 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb" Oct 29 23:53:29.294321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:29.294275 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb"] Oct 29 23:53:29.304287 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:29.304259 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-a4f2db-openshift-default-6cb4596d87-pnhtb"] Oct 29 23:53:29.481168 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:29.481091 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4104e017-b65b-404f-8226-71df8ca5c903" path="/var/lib/kubelet/pods/4104e017-b65b-404f-8226-71df8ca5c903/volumes" Oct 29 23:53:30.797819 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:30.797783 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r"] Oct 29 23:53:30.798286 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:30.798044 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" podUID="44008edf-7b81-4504-adb7-132388744c6e" containerName="istio-proxy" containerID="cri-o://6b8ff8c1bd5c078b830afeae29c52ee31f127004f761d09fb42dcd9965b89e56" gracePeriod=30 Oct 29 23:53:31.767628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:31.767586 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9" podUID="5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:53:31.989497 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:31.989466 5057 patch_prober.go:29] interesting pod/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:53:31.989877 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:31.989517 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" podUID="44008edf-7b81-4504-adb7-132388744c6e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:53:33.789601 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:33.789563 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj" podUID="153fed7b-8844-4d52-a5ad-0b6d8eda52e7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:53:33.913255 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:53:33.913219 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:53:36.195971 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.195941 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:53:36.278532 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.278497 5057 generic.go:356] "Generic (PLEG): container finished" podID="44008edf-7b81-4504-adb7-132388744c6e" containerID="6b8ff8c1bd5c078b830afeae29c52ee31f127004f761d09fb42dcd9965b89e56" exitCode=0 Oct 29 23:53:36.278701 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.278561 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" Oct 29 23:53:36.278701 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.278581 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" event={"ID":"44008edf-7b81-4504-adb7-132388744c6e","Type":"ContainerDied","Data":"6b8ff8c1bd5c078b830afeae29c52ee31f127004f761d09fb42dcd9965b89e56"} Oct 29 23:53:36.278701 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.278680 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r" event={"ID":"44008edf-7b81-4504-adb7-132388744c6e","Type":"ContainerDied","Data":"ea3fffd09860dda1343ee4b5a3ddc88bee53c058a4170f044d17c9f06745a2ff"} Oct 29 23:53:36.278856 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.278707 5057 scope.go:122] "RemoveContainer" containerID="6b8ff8c1bd5c078b830afeae29c52ee31f127004f761d09fb42dcd9965b89e56" Oct 29 23:53:36.288597 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.288573 5057 scope.go:122] "RemoveContainer" containerID="286faffdb87fcb34fbf9536fc0e64fb232b29f1119fc5baed68bdd620efdf3f6" Oct 29 23:53:36.296725 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.296698 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-workload-certs\") pod \"44008edf-7b81-4504-adb7-132388744c6e\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " Oct 29 23:53:36.296873 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.296777 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44008edf-7b81-4504-adb7-132388744c6e-kube-api-access-54pdq\" (UniqueName: \"kubernetes.io/projected/44008edf-7b81-4504-adb7-132388744c6e-kube-api-access-54pdq\") pod \"44008edf-7b81-4504-adb7-132388744c6e\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " Oct 29 23:53:36.296938 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.296913 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44008edf-7b81-4504-adb7-132388744c6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44008edf-7b81-4504-adb7-132388744c6e-istio-podinfo\") pod \"44008edf-7b81-4504-adb7-132388744c6e\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " Oct 29 23:53:36.297007 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.296960 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44008edf-7b81-4504-adb7-132388744c6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44008edf-7b81-4504-adb7-132388744c6e-istiod-ca-cert\") pod \"44008edf-7b81-4504-adb7-132388744c6e\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " Oct 29 23:53:36.297063 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.297002 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-istio-envoy\") pod \"44008edf-7b81-4504-adb7-132388744c6e\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " Oct 29 23:53:36.297063 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.297001 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-workload-certs" pod "44008edf-7b81-4504-adb7-132388744c6e" (UID: "44008edf-7b81-4504-adb7-132388744c6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:36.297063 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.297040 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-credential-socket\") pod \"44008edf-7b81-4504-adb7-132388744c6e\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " Oct 29 23:53:36.297265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.297100 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-workload-socket\") pod \"44008edf-7b81-4504-adb7-132388744c6e\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " Oct 29 23:53:36.297265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.297145 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-istio-data\") pod \"44008edf-7b81-4504-adb7-132388744c6e\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " Oct 29 23:53:36.297387 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.297267 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44008edf-7b81-4504-adb7-132388744c6e-istio-token\" (UniqueName: \"kubernetes.io/projected/44008edf-7b81-4504-adb7-132388744c6e-istio-token\") pod \"44008edf-7b81-4504-adb7-132388744c6e\" (UID: \"44008edf-7b81-4504-adb7-132388744c6e\") " Oct 29 23:53:36.297637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.297610 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-workload-socket" pod "44008edf-7b81-4504-adb7-132388744c6e" (UID: "44008edf-7b81-4504-adb7-132388744c6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:36.297637 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.297619 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-credential-socket" pod "44008edf-7b81-4504-adb7-132388744c6e" (UID: "44008edf-7b81-4504-adb7-132388744c6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:36.297851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.297826 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:36.297923 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.297851 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-istio-data" pod "44008edf-7b81-4504-adb7-132388744c6e" (UID: "44008edf-7b81-4504-adb7-132388744c6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:36.297923 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.297859 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:36.297923 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.297875 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:36.298162 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.298116 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44008edf-7b81-4504-adb7-132388744c6e-istiod-ca-cert" pod "44008edf-7b81-4504-adb7-132388744c6e" (UID: "44008edf-7b81-4504-adb7-132388744c6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:53:36.298680 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.298658 5057 scope.go:122] "RemoveContainer" containerID="6b8ff8c1bd5c078b830afeae29c52ee31f127004f761d09fb42dcd9965b89e56" Oct 29 23:53:36.299046 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:53:36.299019 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b8ff8c1bd5c078b830afeae29c52ee31f127004f761d09fb42dcd9965b89e56\": container with ID starting with 6b8ff8c1bd5c078b830afeae29c52ee31f127004f761d09fb42dcd9965b89e56 not found: ID does not exist" containerID="6b8ff8c1bd5c078b830afeae29c52ee31f127004f761d09fb42dcd9965b89e56" Oct 29 23:53:36.299133 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.299059 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b8ff8c1bd5c078b830afeae29c52ee31f127004f761d09fb42dcd9965b89e56"} err="failed to get container status \"6b8ff8c1bd5c078b830afeae29c52ee31f127004f761d09fb42dcd9965b89e56\": rpc error: code = NotFound desc = could not find container \"6b8ff8c1bd5c078b830afeae29c52ee31f127004f761d09fb42dcd9965b89e56\": container with ID starting with 6b8ff8c1bd5c078b830afeae29c52ee31f127004f761d09fb42dcd9965b89e56 not found: ID does not exist" Oct 29 23:53:36.299133 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.299084 5057 scope.go:122] "RemoveContainer" containerID="286faffdb87fcb34fbf9536fc0e64fb232b29f1119fc5baed68bdd620efdf3f6" Oct 29 23:53:36.299452 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:53:36.299419 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"286faffdb87fcb34fbf9536fc0e64fb232b29f1119fc5baed68bdd620efdf3f6\": container with ID starting with 286faffdb87fcb34fbf9536fc0e64fb232b29f1119fc5baed68bdd620efdf3f6 not found: ID does not exist" containerID="286faffdb87fcb34fbf9536fc0e64fb232b29f1119fc5baed68bdd620efdf3f6" Oct 29 23:53:36.299536 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.299460 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"286faffdb87fcb34fbf9536fc0e64fb232b29f1119fc5baed68bdd620efdf3f6"} err="failed to get container status \"286faffdb87fcb34fbf9536fc0e64fb232b29f1119fc5baed68bdd620efdf3f6\": rpc error: code = NotFound desc = could not find container \"286faffdb87fcb34fbf9536fc0e64fb232b29f1119fc5baed68bdd620efdf3f6\": container with ID starting with 286faffdb87fcb34fbf9536fc0e64fb232b29f1119fc5baed68bdd620efdf3f6 not found: ID does not exist" Oct 29 23:53:36.300120 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.300103 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44008edf-7b81-4504-adb7-132388744c6e-kube-api-access-54pdq" pod "44008edf-7b81-4504-adb7-132388744c6e" (UID: "44008edf-7b81-4504-adb7-132388744c6e"). InnerVolumeSpecName "kube-api-access-54pdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:36.300329 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.300309 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44008edf-7b81-4504-adb7-132388744c6e-istio-token" pod "44008edf-7b81-4504-adb7-132388744c6e" (UID: "44008edf-7b81-4504-adb7-132388744c6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:36.300474 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.300454 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44008edf-7b81-4504-adb7-132388744c6e-istio-podinfo" pod "44008edf-7b81-4504-adb7-132388744c6e" (UID: "44008edf-7b81-4504-adb7-132388744c6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:53:36.300682 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.300664 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-istio-envoy" pod "44008edf-7b81-4504-adb7-132388744c6e" (UID: "44008edf-7b81-4504-adb7-132388744c6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:36.399366 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.399325 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44008edf-7b81-4504-adb7-132388744c6e-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:36.399366 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.399360 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54pdq\" (UniqueName: \"kubernetes.io/projected/44008edf-7b81-4504-adb7-132388744c6e-kube-api-access-54pdq\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:36.399366 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.399371 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44008edf-7b81-4504-adb7-132388744c6e-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:36.399701 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.399381 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44008edf-7b81-4504-adb7-132388744c6e-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:36.399701 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.399390 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:36.399701 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.399398 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44008edf-7b81-4504-adb7-132388744c6e-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:36.585853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.585418 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j"] Oct 29 23:53:36.585853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.585668 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" podUID="9b1937f5-1a2b-4a0d-a42a-636c5b6aafca" containerName="maas-api" containerID="cri-o://e4a1326be837693741933bf2487aa87e45fb25fc07cf0c111420b8f515cc4d6f" gracePeriod=30 Oct 29 23:53:36.689431 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.689401 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r"] Oct 29 23:53:36.693728 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.693693 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a84df8-openshift-default-967bbd69c-lnj8r"] Oct 29 23:53:36.994053 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:36.993739 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" Oct 29 23:53:37.114596 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.114507 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-openshift-service-ca\") pod \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\" (UID: \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\") " Oct 29 23:53:37.114596 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.114573 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-metrics-certs\") pod \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\" (UID: \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\") " Oct 29 23:53:37.114861 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.114643 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-kube-api-access-hsp9r\" (UniqueName: \"kubernetes.io/projected/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-kube-api-access-hsp9r\") pod \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\" (UID: \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\") " Oct 29 23:53:37.114861 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.114678 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-maas-api-tls\") pod \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\" (UID: \"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca\") " Oct 29 23:53:37.115089 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.115066 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-openshift-service-ca" pod "9b1937f5-1a2b-4a0d-a42a-636c5b6aafca" (UID: "9b1937f5-1a2b-4a0d-a42a-636c5b6aafca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:53:37.115242 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.115222 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:37.117455 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.117432 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-metrics-certs" pod "9b1937f5-1a2b-4a0d-a42a-636c5b6aafca" (UID: "9b1937f5-1a2b-4a0d-a42a-636c5b6aafca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:53:37.117588 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.117515 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-kube-api-access-hsp9r" pod "9b1937f5-1a2b-4a0d-a42a-636c5b6aafca" (UID: "9b1937f5-1a2b-4a0d-a42a-636c5b6aafca"). InnerVolumeSpecName "kube-api-access-hsp9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:37.117715 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.117652 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-maas-api-tls" pod "9b1937f5-1a2b-4a0d-a42a-636c5b6aafca" (UID: "9b1937f5-1a2b-4a0d-a42a-636c5b6aafca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:53:37.216411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.216377 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:37.216411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.216407 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsp9r\" (UniqueName: \"kubernetes.io/projected/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-kube-api-access-hsp9r\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:37.216411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.216418 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:37.287805 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.287768 5057 generic.go:356] "Generic (PLEG): container finished" podID="9b1937f5-1a2b-4a0d-a42a-636c5b6aafca" containerID="e4a1326be837693741933bf2487aa87e45fb25fc07cf0c111420b8f515cc4d6f" exitCode=0 Oct 29 23:53:37.288009 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.287815 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" Oct 29 23:53:37.288009 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.287820 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" event={"ID":"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca","Type":"ContainerDied","Data":"e4a1326be837693741933bf2487aa87e45fb25fc07cf0c111420b8f515cc4d6f"} Oct 29 23:53:37.288009 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.287875 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j" event={"ID":"9b1937f5-1a2b-4a0d-a42a-636c5b6aafca","Type":"ContainerDied","Data":"121ac122f544f363ec63554f80acb55a271cd77c001babbde7803831397d5229"} Oct 29 23:53:37.288009 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.287903 5057 scope.go:122] "RemoveContainer" containerID="e4a1326be837693741933bf2487aa87e45fb25fc07cf0c111420b8f515cc4d6f" Oct 29 23:53:37.298086 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.298066 5057 scope.go:122] "RemoveContainer" containerID="e4a1326be837693741933bf2487aa87e45fb25fc07cf0c111420b8f515cc4d6f" Oct 29 23:53:37.298346 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:53:37.298326 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4a1326be837693741933bf2487aa87e45fb25fc07cf0c111420b8f515cc4d6f\": container with ID starting with e4a1326be837693741933bf2487aa87e45fb25fc07cf0c111420b8f515cc4d6f not found: ID does not exist" containerID="e4a1326be837693741933bf2487aa87e45fb25fc07cf0c111420b8f515cc4d6f" Oct 29 23:53:37.298412 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.298355 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a1326be837693741933bf2487aa87e45fb25fc07cf0c111420b8f515cc4d6f"} err="failed to get container status \"e4a1326be837693741933bf2487aa87e45fb25fc07cf0c111420b8f515cc4d6f\": rpc error: code = NotFound desc = could not find container \"e4a1326be837693741933bf2487aa87e45fb25fc07cf0c111420b8f515cc4d6f\": container with ID starting with e4a1326be837693741933bf2487aa87e45fb25fc07cf0c111420b8f515cc4d6f not found: ID does not exist" Oct 29 23:53:37.346858 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.346249 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j"] Oct 29 23:53:37.350162 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.350135 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-383f0e-57b679c6f5-jxp2j"] Oct 29 23:53:37.483423 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.483384 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44008edf-7b81-4504-adb7-132388744c6e" path="/var/lib/kubelet/pods/44008edf-7b81-4504-adb7-132388744c6e/volumes" Oct 29 23:53:37.484036 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:37.484016 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b1937f5-1a2b-4a0d-a42a-636c5b6aafca" path="/var/lib/kubelet/pods/9b1937f5-1a2b-4a0d-a42a-636c5b6aafca/volumes" Oct 29 23:53:41.093945 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.093598 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj" Oct 29 23:53:41.155230 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.154578 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9" Oct 29 23:53:41.174430 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.174400 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153fed7b-8844-4d52-a5ad-0b6d8eda52e7-kube-api-access-gm7wq\" (UniqueName: \"kubernetes.io/projected/153fed7b-8844-4d52-a5ad-0b6d8eda52e7-kube-api-access-gm7wq\") pod \"153fed7b-8844-4d52-a5ad-0b6d8eda52e7\" (UID: \"153fed7b-8844-4d52-a5ad-0b6d8eda52e7\") " Oct 29 23:53:41.174640 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.174623 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/153fed7b-8844-4d52-a5ad-0b6d8eda52e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153fed7b-8844-4d52-a5ad-0b6d8eda52e7-plugins-config-volume\") pod \"153fed7b-8844-4d52-a5ad-0b6d8eda52e7\" (UID: \"153fed7b-8844-4d52-a5ad-0b6d8eda52e7\") " Oct 29 23:53:41.175121 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.175094 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153fed7b-8844-4d52-a5ad-0b6d8eda52e7-plugins-config-volume" pod "153fed7b-8844-4d52-a5ad-0b6d8eda52e7" (UID: "153fed7b-8844-4d52-a5ad-0b6d8eda52e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:53:41.177136 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.177114 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153fed7b-8844-4d52-a5ad-0b6d8eda52e7-kube-api-access-gm7wq" pod "153fed7b-8844-4d52-a5ad-0b6d8eda52e7" (UID: "153fed7b-8844-4d52-a5ad-0b6d8eda52e7"). InnerVolumeSpecName "kube-api-access-gm7wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:41.275969 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.275934 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b-plugins-config-volume\") pod \"5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b\" (UID: \"5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b\") " Oct 29 23:53:41.276137 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.276063 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b-kube-api-access-shz7d\" (UniqueName: \"kubernetes.io/projected/5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b-kube-api-access-shz7d\") pod \"5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b\" (UID: \"5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b\") " Oct 29 23:53:41.276409 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.276382 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b-plugins-config-volume" pod "5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b" (UID: "5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:53:41.276461 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.276391 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm7wq\" (UniqueName: \"kubernetes.io/projected/153fed7b-8844-4d52-a5ad-0b6d8eda52e7-kube-api-access-gm7wq\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:41.276461 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.276449 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153fed7b-8844-4d52-a5ad-0b6d8eda52e7-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:41.278338 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.278316 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b-kube-api-access-shz7d" pod "5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b" (UID: "5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b"). InnerVolumeSpecName "kube-api-access-shz7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:41.316020 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.315947 5057 generic.go:356] "Generic (PLEG): container finished" podID="5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b" containerID="9c8f22e461d7d5785f7faa0e92c98a58940a94c46624e54bc047b508bb089055" exitCode=0 Oct 29 23:53:41.316020 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.316010 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9" Oct 29 23:53:41.316243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.316031 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9" event={"ID":"5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b","Type":"ContainerDied","Data":"9c8f22e461d7d5785f7faa0e92c98a58940a94c46624e54bc047b508bb089055"} Oct 29 23:53:41.316243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.316113 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9" event={"ID":"5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b","Type":"ContainerDied","Data":"5db39611224dbfa8e4e337209d97e5a4cd9aa7ed7291d5923cbabf0d67704167"} Oct 29 23:53:41.316243 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.316135 5057 scope.go:122] "RemoveContainer" containerID="9c8f22e461d7d5785f7faa0e92c98a58940a94c46624e54bc047b508bb089055" Oct 29 23:53:41.317585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.317546 5057 generic.go:356] "Generic (PLEG): container finished" podID="153fed7b-8844-4d52-a5ad-0b6d8eda52e7" containerID="ced5fbebedd978b16c23844f46cf1105fd2c4249a979349af808685c0e2c6b76" exitCode=0 Oct 29 23:53:41.317727 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.317602 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj" Oct 29 23:53:41.317727 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.317716 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj" event={"ID":"153fed7b-8844-4d52-a5ad-0b6d8eda52e7","Type":"ContainerDied","Data":"ced5fbebedd978b16c23844f46cf1105fd2c4249a979349af808685c0e2c6b76"} Oct 29 23:53:41.317851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.317747 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj" event={"ID":"153fed7b-8844-4d52-a5ad-0b6d8eda52e7","Type":"ContainerDied","Data":"5cd02d375d9e555072f7a40a271a728c090f7b73d7988c6fd8a1fd2cb41e9377"} Oct 29 23:53:41.355135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.352606 5057 scope.go:122] "RemoveContainer" containerID="9c8f22e461d7d5785f7faa0e92c98a58940a94c46624e54bc047b508bb089055" Oct 29 23:53:41.355135 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:53:41.353025 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c8f22e461d7d5785f7faa0e92c98a58940a94c46624e54bc047b508bb089055\": container with ID starting with 9c8f22e461d7d5785f7faa0e92c98a58940a94c46624e54bc047b508bb089055 not found: ID does not exist" containerID="9c8f22e461d7d5785f7faa0e92c98a58940a94c46624e54bc047b508bb089055" Oct 29 23:53:41.355135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.353063 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8f22e461d7d5785f7faa0e92c98a58940a94c46624e54bc047b508bb089055"} err="failed to get container status \"9c8f22e461d7d5785f7faa0e92c98a58940a94c46624e54bc047b508bb089055\": rpc error: code = NotFound desc = could not find container \"9c8f22e461d7d5785f7faa0e92c98a58940a94c46624e54bc047b508bb089055\": container with ID starting with 9c8f22e461d7d5785f7faa0e92c98a58940a94c46624e54bc047b508bb089055 not found: ID does not exist" Oct 29 23:53:41.355135 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.353090 5057 scope.go:122] "RemoveContainer" containerID="ced5fbebedd978b16c23844f46cf1105fd2c4249a979349af808685c0e2c6b76" Oct 29 23:53:41.367112 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.367081 5057 scope.go:122] "RemoveContainer" containerID="ced5fbebedd978b16c23844f46cf1105fd2c4249a979349af808685c0e2c6b76" Oct 29 23:53:41.367512 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:53:41.367489 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ced5fbebedd978b16c23844f46cf1105fd2c4249a979349af808685c0e2c6b76\": container with ID starting with ced5fbebedd978b16c23844f46cf1105fd2c4249a979349af808685c0e2c6b76 not found: ID does not exist" containerID="ced5fbebedd978b16c23844f46cf1105fd2c4249a979349af808685c0e2c6b76" Oct 29 23:53:41.367594 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.367523 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced5fbebedd978b16c23844f46cf1105fd2c4249a979349af808685c0e2c6b76"} err="failed to get container status \"ced5fbebedd978b16c23844f46cf1105fd2c4249a979349af808685c0e2c6b76\": rpc error: code = NotFound desc = could not find container \"ced5fbebedd978b16c23844f46cf1105fd2c4249a979349af808685c0e2c6b76\": container with ID starting with ced5fbebedd978b16c23844f46cf1105fd2c4249a979349af808685c0e2c6b76 not found: ID does not exist" Oct 29 23:53:41.369673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.369633 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj"] Oct 29 23:53:41.373241 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.373216 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a84df8-6565949554-pvkvj"] Oct 29 23:53:41.377237 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.377211 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:41.377237 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.377233 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shz7d\" (UniqueName: \"kubernetes.io/projected/5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b-kube-api-access-shz7d\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:41.407698 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.407662 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9"] Oct 29 23:53:41.412139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.412115 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a84df8-847b8f6575-fbvn9"] Oct 29 23:53:41.483202 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.483143 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="153fed7b-8844-4d52-a5ad-0b6d8eda52e7" path="/var/lib/kubelet/pods/153fed7b-8844-4d52-a5ad-0b6d8eda52e7/volumes" Oct 29 23:53:41.483592 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:41.483571 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b" path="/var/lib/kubelet/pods/5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b/volumes" Oct 29 23:53:42.108928 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:42.108895 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:53:42.326355 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:42.326276 5057 generic.go:356] "Generic (PLEG): container finished" podID="d032ce48-7955-47e5-a74b-9af1ef81ed17" containerID="fd497a9ecdc6abe0f1405a9b283b31a198da24a6f58f8d93d1b99cb2d762063a" exitCode=0 Oct 29 23:53:42.326529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:42.326351 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" event={"ID":"d032ce48-7955-47e5-a74b-9af1ef81ed17","Type":"ContainerDied","Data":"fd497a9ecdc6abe0f1405a9b283b31a198da24a6f58f8d93d1b99cb2d762063a"} Oct 29 23:53:42.326843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:42.326815 5057 scope.go:122] "RemoveContainer" containerID="fd497a9ecdc6abe0f1405a9b283b31a198da24a6f58f8d93d1b99cb2d762063a" Oct 29 23:53:42.328524 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:42.328487 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:53:42.328600 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:42.328582 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:53:42.328656 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:42.328627 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 29 23:53:42.816869 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:42.816831 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:53:43.352879 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:43.352839 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" event={"ID":"d032ce48-7955-47e5-a74b-9af1ef81ed17","Type":"ContainerStarted","Data":"684dcefb60d3ae93ab7eaff185b9412075cd308d54da026eb9e1e9da8ee1462f"} Oct 29 23:53:43.447628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:43.447594 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:53:43.449269 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:43.449236 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 29 23:53:43.449546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:43.449520 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" podUID="d032ce48-7955-47e5-a74b-9af1ef81ed17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 29 23:53:44.381543 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:44.381514 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r"] Oct 29 23:53:44.382240 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:44.382208 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" podUID="e5ee3546-b9fa-49f9-a7b1-c110932ebb10" containerName="istio-proxy" containerID="cri-o://f43af5c9cd246f7adfbc1d4ad3d08d0b92b7a172751f9a186c7e9aefec4191c7" gracePeriod=30 Oct 29 23:53:44.457046 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:44.456677 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 29 23:53:44.457046 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:44.456741 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" podUID="d032ce48-7955-47e5-a74b-9af1ef81ed17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 29 23:53:44.987323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:44.986398 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49"] Oct 29 23:53:44.987323 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:44.986612 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" podUID="f255ae1a-010b-4f50-9414-5b09b507c085" containerName="istio-proxy" containerID="cri-o://4942e4c8f4e388ec45f65eb53e1f8a53221dbc5e62fd4660e2abde9e34e227aa" gracePeriod=30 Oct 29 23:53:45.450676 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:45.450639 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 29 23:53:45.451354 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:45.451316 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" podUID="d032ce48-7955-47e5-a74b-9af1ef81ed17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 29 23:53:46.448261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:46.447704 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 29 23:53:46.448261 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:46.447771 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" podUID="d032ce48-7955-47e5-a74b-9af1ef81ed17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 29 23:53:47.452706 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:47.452656 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 29 23:53:47.453271 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:47.452715 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" podUID="d032ce48-7955-47e5-a74b-9af1ef81ed17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 29 23:53:48.445472 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:48.445432 5057 patch_prober.go:29] interesting pod/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 29 23:53:48.445685 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:48.445499 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" podUID="d032ce48-7955-47e5-a74b-9af1ef81ed17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 29 23:53:49.448921 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.448894 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:53:49.449405 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.449391 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:53:49.450053 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.450033 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:53:49.664000 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.663905 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:53:49.817359 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.817324 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-data\") pod \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " Oct 29 23:53:49.817576 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.817395 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-token\" (UniqueName: \"kubernetes.io/projected/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-token\") pod \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " Oct 29 23:53:49.817576 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.817451 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-workload-certs\") pod \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " Oct 29 23:53:49.817576 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.817476 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-envoy\") pod \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " Oct 29 23:53:49.817770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.817645 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istiod-ca-cert\") pod \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " Oct 29 23:53:49.817770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.817679 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-podinfo\") pod \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " Oct 29 23:53:49.817770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.817759 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-kube-api-access-z79dk\" (UniqueName: \"kubernetes.io/projected/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-kube-api-access-z79dk\") pod \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " Oct 29 23:53:49.817943 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.817798 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-credential-socket\") pod \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " Oct 29 23:53:49.817943 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.817686 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-data" pod "e5ee3546-b9fa-49f9-a7b1-c110932ebb10" (UID: "e5ee3546-b9fa-49f9-a7b1-c110932ebb10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:49.817943 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.817881 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-workload-socket\") pod \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\" (UID: \"e5ee3546-b9fa-49f9-a7b1-c110932ebb10\") " Oct 29 23:53:49.817943 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.817810 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-workload-certs" pod "e5ee3546-b9fa-49f9-a7b1-c110932ebb10" (UID: "e5ee3546-b9fa-49f9-a7b1-c110932ebb10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:49.818171 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.818035 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istiod-ca-cert" pod "e5ee3546-b9fa-49f9-a7b1-c110932ebb10" (UID: "e5ee3546-b9fa-49f9-a7b1-c110932ebb10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:53:49.818171 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.818053 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-credential-socket" pod "e5ee3546-b9fa-49f9-a7b1-c110932ebb10" (UID: "e5ee3546-b9fa-49f9-a7b1-c110932ebb10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:49.818291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.818233 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-workload-socket" pod "e5ee3546-b9fa-49f9-a7b1-c110932ebb10" (UID: "e5ee3546-b9fa-49f9-a7b1-c110932ebb10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:49.818537 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.818517 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:49.818626 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.818538 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:49.818626 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.818549 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:49.818626 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.818557 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:49.818626 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.818565 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:49.819984 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.819954 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-kube-api-access-z79dk" pod "e5ee3546-b9fa-49f9-a7b1-c110932ebb10" (UID: "e5ee3546-b9fa-49f9-a7b1-c110932ebb10"). InnerVolumeSpecName "kube-api-access-z79dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:49.820118 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.820095 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-envoy" pod "e5ee3546-b9fa-49f9-a7b1-c110932ebb10" (UID: "e5ee3546-b9fa-49f9-a7b1-c110932ebb10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:49.820281 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.820256 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-podinfo" pod "e5ee3546-b9fa-49f9-a7b1-c110932ebb10" (UID: "e5ee3546-b9fa-49f9-a7b1-c110932ebb10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:53:49.820281 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.820275 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-token" pod "e5ee3546-b9fa-49f9-a7b1-c110932ebb10" (UID: "e5ee3546-b9fa-49f9-a7b1-c110932ebb10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:49.919730 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.919637 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z79dk\" (UniqueName: \"kubernetes.io/projected/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-kube-api-access-z79dk\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:49.919730 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.919671 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:49.919730 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.919685 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:49.919730 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:49.919696 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ee3546-b9fa-49f9-a7b1-c110932ebb10-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:50.259629 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.259600 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:53:50.324708 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.324673 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-istio-envoy\") pod \"f255ae1a-010b-4f50-9414-5b09b507c085\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " Oct 29 23:53:50.324871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.324747 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f255ae1a-010b-4f50-9414-5b09b507c085-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f255ae1a-010b-4f50-9414-5b09b507c085-istiod-ca-cert\") pod \"f255ae1a-010b-4f50-9414-5b09b507c085\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " Oct 29 23:53:50.324871 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.324803 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f255ae1a-010b-4f50-9414-5b09b507c085-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f255ae1a-010b-4f50-9414-5b09b507c085-istio-podinfo\") pod \"f255ae1a-010b-4f50-9414-5b09b507c085\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " Oct 29 23:53:50.324997 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.324878 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f255ae1a-010b-4f50-9414-5b09b507c085-kube-api-access-r2jm4\" (UniqueName: \"kubernetes.io/projected/f255ae1a-010b-4f50-9414-5b09b507c085-kube-api-access-r2jm4\") pod \"f255ae1a-010b-4f50-9414-5b09b507c085\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " Oct 29 23:53:50.324997 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.324906 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-workload-socket\") pod \"f255ae1a-010b-4f50-9414-5b09b507c085\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " Oct 29 23:53:50.325119 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.325006 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-istio-data\") pod \"f255ae1a-010b-4f50-9414-5b09b507c085\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " Oct 29 23:53:50.325119 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.325053 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-credential-socket\") pod \"f255ae1a-010b-4f50-9414-5b09b507c085\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " Oct 29 23:53:50.325119 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.325095 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-workload-certs\") pod \"f255ae1a-010b-4f50-9414-5b09b507c085\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " Oct 29 23:53:50.326327 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.325161 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f255ae1a-010b-4f50-9414-5b09b507c085-istio-token\" (UniqueName: \"kubernetes.io/projected/f255ae1a-010b-4f50-9414-5b09b507c085-istio-token\") pod \"f255ae1a-010b-4f50-9414-5b09b507c085\" (UID: \"f255ae1a-010b-4f50-9414-5b09b507c085\") " Oct 29 23:53:50.326327 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.325181 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f255ae1a-010b-4f50-9414-5b09b507c085-istiod-ca-cert" pod "f255ae1a-010b-4f50-9414-5b09b507c085" (UID: "f255ae1a-010b-4f50-9414-5b09b507c085"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:53:50.326327 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.325542 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-workload-certs" pod "f255ae1a-010b-4f50-9414-5b09b507c085" (UID: "f255ae1a-010b-4f50-9414-5b09b507c085"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:50.326327 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.325676 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:50.326327 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.325698 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f255ae1a-010b-4f50-9414-5b09b507c085-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:50.326327 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.325699 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-istio-data" pod "f255ae1a-010b-4f50-9414-5b09b507c085" (UID: "f255ae1a-010b-4f50-9414-5b09b507c085"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:50.326327 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.325762 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-workload-socket" pod "f255ae1a-010b-4f50-9414-5b09b507c085" (UID: "f255ae1a-010b-4f50-9414-5b09b507c085"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:50.326327 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.325861 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-credential-socket" pod "f255ae1a-010b-4f50-9414-5b09b507c085" (UID: "f255ae1a-010b-4f50-9414-5b09b507c085"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:50.327450 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.327426 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-istio-envoy" pod "f255ae1a-010b-4f50-9414-5b09b507c085" (UID: "f255ae1a-010b-4f50-9414-5b09b507c085"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:50.327450 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.327439 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f255ae1a-010b-4f50-9414-5b09b507c085-kube-api-access-r2jm4" pod "f255ae1a-010b-4f50-9414-5b09b507c085" (UID: "f255ae1a-010b-4f50-9414-5b09b507c085"). InnerVolumeSpecName "kube-api-access-r2jm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:50.327634 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.327618 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f255ae1a-010b-4f50-9414-5b09b507c085-istio-podinfo" pod "f255ae1a-010b-4f50-9414-5b09b507c085" (UID: "f255ae1a-010b-4f50-9414-5b09b507c085"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:53:50.328026 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.328006 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f255ae1a-010b-4f50-9414-5b09b507c085-istio-token" pod "f255ae1a-010b-4f50-9414-5b09b507c085" (UID: "f255ae1a-010b-4f50-9414-5b09b507c085"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:50.421279 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.421245 5057 generic.go:356] "Generic (PLEG): container finished" podID="f255ae1a-010b-4f50-9414-5b09b507c085" containerID="4942e4c8f4e388ec45f65eb53e1f8a53221dbc5e62fd4660e2abde9e34e227aa" exitCode=0 Oct 29 23:53:50.421431 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.421298 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" Oct 29 23:53:50.421431 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.421326 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" event={"ID":"f255ae1a-010b-4f50-9414-5b09b507c085","Type":"ContainerDied","Data":"4942e4c8f4e388ec45f65eb53e1f8a53221dbc5e62fd4660e2abde9e34e227aa"} Oct 29 23:53:50.421431 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.421376 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49" event={"ID":"f255ae1a-010b-4f50-9414-5b09b507c085","Type":"ContainerDied","Data":"f572f2131e825f22ae0caf37701d14e615791bce802682ff5db0a92aa72d41e3"} Oct 29 23:53:50.421431 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.421403 5057 scope.go:122] "RemoveContainer" containerID="4942e4c8f4e388ec45f65eb53e1f8a53221dbc5e62fd4660e2abde9e34e227aa" Oct 29 23:53:50.422845 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.422823 5057 generic.go:356] "Generic (PLEG): container finished" podID="e5ee3546-b9fa-49f9-a7b1-c110932ebb10" containerID="f43af5c9cd246f7adfbc1d4ad3d08d0b92b7a172751f9a186c7e9aefec4191c7" exitCode=0 Oct 29 23:53:50.422959 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.422912 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" Oct 29 23:53:50.422959 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.422934 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" event={"ID":"e5ee3546-b9fa-49f9-a7b1-c110932ebb10","Type":"ContainerDied","Data":"f43af5c9cd246f7adfbc1d4ad3d08d0b92b7a172751f9a186c7e9aefec4191c7"} Oct 29 23:53:50.423056 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.422990 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r" event={"ID":"e5ee3546-b9fa-49f9-a7b1-c110932ebb10","Type":"ContainerDied","Data":"303e38f689a2f80acdd878d05f48487749c7cee45907d27f8af72e95dd35a92a"} Oct 29 23:53:50.426704 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.426682 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f255ae1a-010b-4f50-9414-5b09b507c085-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:50.426791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.426711 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2jm4\" (UniqueName: \"kubernetes.io/projected/f255ae1a-010b-4f50-9414-5b09b507c085-kube-api-access-r2jm4\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:50.426791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.426729 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:50.426791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.426743 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:50.426791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.426757 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:50.426791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.426771 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f255ae1a-010b-4f50-9414-5b09b507c085-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:50.426791 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.426784 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f255ae1a-010b-4f50-9414-5b09b507c085-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:50.431667 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.431649 5057 scope.go:122] "RemoveContainer" containerID="12f9dddeba1de36d011970ce9fc27ed207ddb84b6c30fd3e37d49eafd3881b41" Oct 29 23:53:50.440916 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.440899 5057 scope.go:122] "RemoveContainer" containerID="4942e4c8f4e388ec45f65eb53e1f8a53221dbc5e62fd4660e2abde9e34e227aa" Oct 29 23:53:50.441220 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:53:50.441168 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4942e4c8f4e388ec45f65eb53e1f8a53221dbc5e62fd4660e2abde9e34e227aa\": container with ID starting with 4942e4c8f4e388ec45f65eb53e1f8a53221dbc5e62fd4660e2abde9e34e227aa not found: ID does not exist" containerID="4942e4c8f4e388ec45f65eb53e1f8a53221dbc5e62fd4660e2abde9e34e227aa" Oct 29 23:53:50.441340 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.441304 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4942e4c8f4e388ec45f65eb53e1f8a53221dbc5e62fd4660e2abde9e34e227aa"} err="failed to get container status \"4942e4c8f4e388ec45f65eb53e1f8a53221dbc5e62fd4660e2abde9e34e227aa\": rpc error: code = NotFound desc = could not find container \"4942e4c8f4e388ec45f65eb53e1f8a53221dbc5e62fd4660e2abde9e34e227aa\": container with ID starting with 4942e4c8f4e388ec45f65eb53e1f8a53221dbc5e62fd4660e2abde9e34e227aa not found: ID does not exist" Oct 29 23:53:50.441422 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.441343 5057 scope.go:122] "RemoveContainer" containerID="12f9dddeba1de36d011970ce9fc27ed207ddb84b6c30fd3e37d49eafd3881b41" Oct 29 23:53:50.441593 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:53:50.441571 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f9dddeba1de36d011970ce9fc27ed207ddb84b6c30fd3e37d49eafd3881b41\": container with ID starting with 12f9dddeba1de36d011970ce9fc27ed207ddb84b6c30fd3e37d49eafd3881b41 not found: ID does not exist" containerID="12f9dddeba1de36d011970ce9fc27ed207ddb84b6c30fd3e37d49eafd3881b41" Oct 29 23:53:50.441686 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.441595 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f9dddeba1de36d011970ce9fc27ed207ddb84b6c30fd3e37d49eafd3881b41"} err="failed to get container status \"12f9dddeba1de36d011970ce9fc27ed207ddb84b6c30fd3e37d49eafd3881b41\": rpc error: code = NotFound desc = could not find container \"12f9dddeba1de36d011970ce9fc27ed207ddb84b6c30fd3e37d49eafd3881b41\": container with ID starting with 12f9dddeba1de36d011970ce9fc27ed207ddb84b6c30fd3e37d49eafd3881b41 not found: ID does not exist" Oct 29 23:53:50.441686 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.441609 5057 scope.go:122] "RemoveContainer" containerID="f43af5c9cd246f7adfbc1d4ad3d08d0b92b7a172751f9a186c7e9aefec4191c7" Oct 29 23:53:50.450234 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.450217 5057 scope.go:122] "RemoveContainer" containerID="f43af5c9cd246f7adfbc1d4ad3d08d0b92b7a172751f9a186c7e9aefec4191c7" Oct 29 23:53:50.450589 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:53:50.450511 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f43af5c9cd246f7adfbc1d4ad3d08d0b92b7a172751f9a186c7e9aefec4191c7\": container with ID starting with f43af5c9cd246f7adfbc1d4ad3d08d0b92b7a172751f9a186c7e9aefec4191c7 not found: ID does not exist" containerID="f43af5c9cd246f7adfbc1d4ad3d08d0b92b7a172751f9a186c7e9aefec4191c7" Oct 29 23:53:50.450589 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.450539 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f43af5c9cd246f7adfbc1d4ad3d08d0b92b7a172751f9a186c7e9aefec4191c7"} err="failed to get container status \"f43af5c9cd246f7adfbc1d4ad3d08d0b92b7a172751f9a186c7e9aefec4191c7\": rpc error: code = NotFound desc = could not find container \"f43af5c9cd246f7adfbc1d4ad3d08d0b92b7a172751f9a186c7e9aefec4191c7\": container with ID starting with f43af5c9cd246f7adfbc1d4ad3d08d0b92b7a172751f9a186c7e9aefec4191c7 not found: ID does not exist" Oct 29 23:53:50.475252 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.475217 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49"] Oct 29 23:53:50.477567 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.477547 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s"] Oct 29 23:53:50.482858 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.482812 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-383f0e-openshift-default-7775b6d45c-2qg49"] Oct 29 23:53:50.522626 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.522540 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r"] Oct 29 23:53:50.533484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:50.533453 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-x4v6r"] Oct 29 23:53:51.483353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:51.483311 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ee3546-b9fa-49f9-a7b1-c110932ebb10" path="/var/lib/kubelet/pods/e5ee3546-b9fa-49f9-a7b1-c110932ebb10/volumes" Oct 29 23:53:51.483810 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:51.483794 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f255ae1a-010b-4f50-9414-5b09b507c085" path="/var/lib/kubelet/pods/f255ae1a-010b-4f50-9414-5b09b507c085/volumes" Oct 29 23:53:52.049096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:52.049057 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv"] Oct 29 23:53:52.436884 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:52.436848 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" podUID="d032ce48-7955-47e5-a74b-9af1ef81ed17" containerName="istio-proxy" containerID="cri-o://684dcefb60d3ae93ab7eaff185b9412075cd308d54da026eb9e1e9da8ee1462f" gracePeriod=30 Oct 29 23:53:57.230946 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.230903 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:53:57.742210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.742164 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:53:57.838299 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.838213 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-envoy\") pod \"d032ce48-7955-47e5-a74b-9af1ef81ed17\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " Oct 29 23:53:57.838299 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.838275 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-podinfo\") pod \"d032ce48-7955-47e5-a74b-9af1ef81ed17\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " Oct 29 23:53:57.838550 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.838347 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-data\") pod \"d032ce48-7955-47e5-a74b-9af1ef81ed17\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " Oct 29 23:53:57.838550 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.838378 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-token\" (UniqueName: \"kubernetes.io/projected/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-token\") pod \"d032ce48-7955-47e5-a74b-9af1ef81ed17\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " Oct 29 23:53:57.838550 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.838464 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d032ce48-7955-47e5-a74b-9af1ef81ed17-kube-api-access-lqgp8\" (UniqueName: \"kubernetes.io/projected/d032ce48-7955-47e5-a74b-9af1ef81ed17-kube-api-access-lqgp8\") pod \"d032ce48-7955-47e5-a74b-9af1ef81ed17\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " Oct 29 23:53:57.838722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.838548 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-credential-socket\") pod \"d032ce48-7955-47e5-a74b-9af1ef81ed17\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " Oct 29 23:53:57.838722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.838649 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-workload-socket\") pod \"d032ce48-7955-47e5-a74b-9af1ef81ed17\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " Oct 29 23:53:57.838722 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.838705 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-data" pod "d032ce48-7955-47e5-a74b-9af1ef81ed17" (UID: "d032ce48-7955-47e5-a74b-9af1ef81ed17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:57.838885 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.838737 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d032ce48-7955-47e5-a74b-9af1ef81ed17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d032ce48-7955-47e5-a74b-9af1ef81ed17-istiod-ca-cert\") pod \"d032ce48-7955-47e5-a74b-9af1ef81ed17\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " Oct 29 23:53:57.838885 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.838767 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-workload-certs\") pod \"d032ce48-7955-47e5-a74b-9af1ef81ed17\" (UID: \"d032ce48-7955-47e5-a74b-9af1ef81ed17\") " Oct 29 23:53:57.838992 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.838904 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-credential-socket" pod "d032ce48-7955-47e5-a74b-9af1ef81ed17" (UID: "d032ce48-7955-47e5-a74b-9af1ef81ed17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:57.839226 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.839163 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d032ce48-7955-47e5-a74b-9af1ef81ed17-istiod-ca-cert" pod "d032ce48-7955-47e5-a74b-9af1ef81ed17" (UID: "d032ce48-7955-47e5-a74b-9af1ef81ed17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:53:57.839483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.839461 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-workload-certs" pod "d032ce48-7955-47e5-a74b-9af1ef81ed17" (UID: "d032ce48-7955-47e5-a74b-9af1ef81ed17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:57.839575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.839179 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:57.839575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.839507 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:57.839915 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.839893 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-workload-socket" pod "d032ce48-7955-47e5-a74b-9af1ef81ed17" (UID: "d032ce48-7955-47e5-a74b-9af1ef81ed17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:57.841413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.841362 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-envoy" pod "d032ce48-7955-47e5-a74b-9af1ef81ed17" (UID: "d032ce48-7955-47e5-a74b-9af1ef81ed17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:57.841413 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.841366 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-podinfo" pod "d032ce48-7955-47e5-a74b-9af1ef81ed17" (UID: "d032ce48-7955-47e5-a74b-9af1ef81ed17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:53:57.841563 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.841500 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d032ce48-7955-47e5-a74b-9af1ef81ed17-kube-api-access-lqgp8" pod "d032ce48-7955-47e5-a74b-9af1ef81ed17" (UID: "d032ce48-7955-47e5-a74b-9af1ef81ed17"). InnerVolumeSpecName "kube-api-access-lqgp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:57.841607 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.841595 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-token" pod "d032ce48-7955-47e5-a74b-9af1ef81ed17" (UID: "d032ce48-7955-47e5-a74b-9af1ef81ed17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:57.940874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.940838 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:57.940874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.940869 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqgp8\" (UniqueName: \"kubernetes.io/projected/d032ce48-7955-47e5-a74b-9af1ef81ed17-kube-api-access-lqgp8\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:57.941083 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.940883 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:57.941083 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.940895 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d032ce48-7955-47e5-a74b-9af1ef81ed17-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:57.941083 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.940905 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:57.941083 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.940912 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:57.941083 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:57.940921 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d032ce48-7955-47e5-a74b-9af1ef81ed17-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:53:58.475768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:58.475734 5057 generic.go:356] "Generic (PLEG): container finished" podID="d032ce48-7955-47e5-a74b-9af1ef81ed17" containerID="684dcefb60d3ae93ab7eaff185b9412075cd308d54da026eb9e1e9da8ee1462f" exitCode=0 Oct 29 23:53:58.476314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:58.475789 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" Oct 29 23:53:58.476314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:58.475816 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" event={"ID":"d032ce48-7955-47e5-a74b-9af1ef81ed17","Type":"ContainerDied","Data":"684dcefb60d3ae93ab7eaff185b9412075cd308d54da026eb9e1e9da8ee1462f"} Oct 29 23:53:58.476314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:58.475869 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv" event={"ID":"d032ce48-7955-47e5-a74b-9af1ef81ed17","Type":"ContainerDied","Data":"e6287e7461693a61f2b039fb5bd4044d8e006c0d2205feee91a263d039a3d796"} Oct 29 23:53:58.476314 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:58.475893 5057 scope.go:122] "RemoveContainer" containerID="684dcefb60d3ae93ab7eaff185b9412075cd308d54da026eb9e1e9da8ee1462f" Oct 29 23:53:58.486411 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:58.486389 5057 scope.go:122] "RemoveContainer" containerID="fd497a9ecdc6abe0f1405a9b283b31a198da24a6f58f8d93d1b99cb2d762063a" Oct 29 23:53:58.495395 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:58.495377 5057 scope.go:122] "RemoveContainer" containerID="684dcefb60d3ae93ab7eaff185b9412075cd308d54da026eb9e1e9da8ee1462f" Oct 29 23:53:58.495684 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:53:58.495661 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684dcefb60d3ae93ab7eaff185b9412075cd308d54da026eb9e1e9da8ee1462f\": container with ID starting with 684dcefb60d3ae93ab7eaff185b9412075cd308d54da026eb9e1e9da8ee1462f not found: ID does not exist" containerID="684dcefb60d3ae93ab7eaff185b9412075cd308d54da026eb9e1e9da8ee1462f" Oct 29 23:53:58.495741 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:58.495693 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684dcefb60d3ae93ab7eaff185b9412075cd308d54da026eb9e1e9da8ee1462f"} err="failed to get container status \"684dcefb60d3ae93ab7eaff185b9412075cd308d54da026eb9e1e9da8ee1462f\": rpc error: code = NotFound desc = could not find container \"684dcefb60d3ae93ab7eaff185b9412075cd308d54da026eb9e1e9da8ee1462f\": container with ID starting with 684dcefb60d3ae93ab7eaff185b9412075cd308d54da026eb9e1e9da8ee1462f not found: ID does not exist" Oct 29 23:53:58.495741 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:58.495712 5057 scope.go:122] "RemoveContainer" containerID="fd497a9ecdc6abe0f1405a9b283b31a198da24a6f58f8d93d1b99cb2d762063a" Oct 29 23:53:58.495956 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:53:58.495937 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd497a9ecdc6abe0f1405a9b283b31a198da24a6f58f8d93d1b99cb2d762063a\": container with ID starting with fd497a9ecdc6abe0f1405a9b283b31a198da24a6f58f8d93d1b99cb2d762063a not found: ID does not exist" containerID="fd497a9ecdc6abe0f1405a9b283b31a198da24a6f58f8d93d1b99cb2d762063a" Oct 29 23:53:58.496018 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:58.495967 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd497a9ecdc6abe0f1405a9b283b31a198da24a6f58f8d93d1b99cb2d762063a"} err="failed to get container status \"fd497a9ecdc6abe0f1405a9b283b31a198da24a6f58f8d93d1b99cb2d762063a\": rpc error: code = NotFound desc = could not find container \"fd497a9ecdc6abe0f1405a9b283b31a198da24a6f58f8d93d1b99cb2d762063a\": container with ID starting with fd497a9ecdc6abe0f1405a9b283b31a198da24a6f58f8d93d1b99cb2d762063a not found: ID does not exist" Oct 29 23:53:58.541657 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:58.541616 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv"] Oct 29 23:53:58.552730 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:58.552701 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e59cd-openshift-default-7df9cdc8bf-rjkjv"] Oct 29 23:53:59.480447 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:53:59.480414 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d032ce48-7955-47e5-a74b-9af1ef81ed17" path="/var/lib/kubelet/pods/d032ce48-7955-47e5-a74b-9af1ef81ed17/volumes" Oct 29 23:54:03.913577 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:54:03.913541 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:54:05.499415 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:05.499378 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" containerID="cri-o://c2644bec3bdd682a4dace47ed469a669ef075214e6a59213b77e3226e04865ba" gracePeriod=45 Oct 29 23:54:05.776106 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:05.776079 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:54:05.825434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:05.825284 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-home\" (UniqueName: \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-home\") pod \"4a56baba-0899-4946-89d0-13db7c5c0242\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " Oct 29 23:54:05.831047 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:05.829313 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-home" pod "4a56baba-0899-4946-89d0-13db7c5c0242" (UID: "4a56baba-0899-4946-89d0-13db7c5c0242"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:05.831047 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:05.829891 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-tmp-dir\") pod \"4a56baba-0899-4946-89d0-13db7c5c0242\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " Oct 29 23:54:05.831047 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:05.829939 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-dshm\") pod \"4a56baba-0899-4946-89d0-13db7c5c0242\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " Oct 29 23:54:05.831047 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:05.829968 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a56baba-0899-4946-89d0-13db7c5c0242-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a56baba-0899-4946-89d0-13db7c5c0242-tls-certs\") pod \"4a56baba-0899-4946-89d0-13db7c5c0242\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " Oct 29 23:54:05.831047 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:05.830072 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-model-cache\") pod \"4a56baba-0899-4946-89d0-13db7c5c0242\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " Oct 29 23:54:05.831047 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:05.830118 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a56baba-0899-4946-89d0-13db7c5c0242-kube-api-access-zkb5h\" (UniqueName: \"kubernetes.io/projected/4a56baba-0899-4946-89d0-13db7c5c0242-kube-api-access-zkb5h\") pod \"4a56baba-0899-4946-89d0-13db7c5c0242\" (UID: \"4a56baba-0899-4946-89d0-13db7c5c0242\") " Oct 29 23:54:05.831047 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:05.830635 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-home\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:05.833342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:05.833129 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-tmp-dir" pod "4a56baba-0899-4946-89d0-13db7c5c0242" (UID: "4a56baba-0899-4946-89d0-13db7c5c0242"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:05.833342 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:05.833305 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-model-cache" pod "4a56baba-0899-4946-89d0-13db7c5c0242" (UID: "4a56baba-0899-4946-89d0-13db7c5c0242"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:05.834280 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:05.834248 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a56baba-0899-4946-89d0-13db7c5c0242-kube-api-access-zkb5h" pod "4a56baba-0899-4946-89d0-13db7c5c0242" (UID: "4a56baba-0899-4946-89d0-13db7c5c0242"). InnerVolumeSpecName "kube-api-access-zkb5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:05.834918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:05.834886 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-dshm" pod "4a56baba-0899-4946-89d0-13db7c5c0242" (UID: "4a56baba-0899-4946-89d0-13db7c5c0242"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:05.851761 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:05.851526 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a56baba-0899-4946-89d0-13db7c5c0242-tls-certs" pod "4a56baba-0899-4946-89d0-13db7c5c0242" (UID: "4a56baba-0899-4946-89d0-13db7c5c0242"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:54:05.931557 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:05.931517 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-model-cache\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:05.931557 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:05.931559 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkb5h\" (UniqueName: \"kubernetes.io/projected/4a56baba-0899-4946-89d0-13db7c5c0242-kube-api-access-zkb5h\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:05.931813 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:05.931575 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-tmp-dir\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:05.931813 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:05.931590 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a56baba-0899-4946-89d0-13db7c5c0242-dshm\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:05.931813 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:05.931602 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a56baba-0899-4946-89d0-13db7c5c0242-tls-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:06.164564 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:06.164475 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr"] Oct 29 23:54:06.200370 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:06.200337 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2"] Oct 29 23:54:06.527166 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:06.527124 5057 generic.go:356] "Generic (PLEG): container finished" podID="4a56baba-0899-4946-89d0-13db7c5c0242" containerID="c2644bec3bdd682a4dace47ed469a669ef075214e6a59213b77e3226e04865ba" exitCode=0 Oct 29 23:54:06.527694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:06.527178 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" event={"ID":"4a56baba-0899-4946-89d0-13db7c5c0242","Type":"ContainerDied","Data":"c2644bec3bdd682a4dace47ed469a669ef075214e6a59213b77e3226e04865ba"} Oct 29 23:54:06.527694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:06.527179 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" Oct 29 23:54:06.527694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:06.527253 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s" event={"ID":"4a56baba-0899-4946-89d0-13db7c5c0242","Type":"ContainerDied","Data":"c6c98f01045aef1a65629f7deb867b4560841222b7a312dc106ed91907ddf704"} Oct 29 23:54:06.527694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:06.527282 5057 scope.go:122] "RemoveContainer" containerID="c2644bec3bdd682a4dace47ed469a669ef075214e6a59213b77e3226e04865ba" Oct 29 23:54:06.539447 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:06.539414 5057 scope.go:122] "RemoveContainer" containerID="c2644bec3bdd682a4dace47ed469a669ef075214e6a59213b77e3226e04865ba" Oct 29 23:54:06.539724 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:54:06.539703 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2644bec3bdd682a4dace47ed469a669ef075214e6a59213b77e3226e04865ba\": container with ID starting with c2644bec3bdd682a4dace47ed469a669ef075214e6a59213b77e3226e04865ba not found: ID does not exist" containerID="c2644bec3bdd682a4dace47ed469a669ef075214e6a59213b77e3226e04865ba" Oct 29 23:54:06.539816 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:06.539732 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2644bec3bdd682a4dace47ed469a669ef075214e6a59213b77e3226e04865ba"} err="failed to get container status \"c2644bec3bdd682a4dace47ed469a669ef075214e6a59213b77e3226e04865ba\": rpc error: code = NotFound desc = could not find container \"c2644bec3bdd682a4dace47ed469a669ef075214e6a59213b77e3226e04865ba\": container with ID starting with c2644bec3bdd682a4dace47ed469a669ef075214e6a59213b77e3226e04865ba not found: ID does not exist" Oct 29 23:54:06.578717 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:06.578686 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s"] Oct 29 23:54:06.584573 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:06.584544 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/distinct-2-w5-2077b4-kserve-7d4b6f98d9-2vk4s"] Oct 29 23:54:07.480523 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:07.480491 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" path="/var/lib/kubelet/pods/4a56baba-0899-4946-89d0-13db7c5c0242/volumes" Oct 29 23:54:11.164891 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:11.164825 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr" podUID="87936d4c-796b-4176-b54a-4b520e0b7ce5" containerName="payload-processing" containerID="cri-o://63afa5a3a93d6f74b869d00e48fb75c41899dd8f639a9117fe3cff87b35472ce" gracePeriod=25 Oct 29 23:54:11.200787 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:11.200728 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2" podUID="e3a0a5b1-c514-4685-a97d-c75ea6882e82" containerName="payload-pre-processing" containerID="cri-o://c4ce3d9796ce9730a641d772c9bd023d34ec70de5091cb8b3945920278671b4f" gracePeriod=25 Oct 29 23:54:11.972803 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:11.972766 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2" podUID="e3a0a5b1-c514-4685-a97d-c75ea6882e82" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:54:11.973031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:11.972776 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr" podUID="87936d4c-796b-4176-b54a-4b520e0b7ce5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:54:11.993610 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:11.993574 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x"] Oct 29 23:54:11.993846 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:11.993817 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" podUID="fe0b52e3-7076-420c-b899-4a8b4a3db390" containerName="istio-proxy" containerID="cri-o://3eedf5d6f7cb7cca0d2451364457707723a72f680ea0c729d6d43fcd3fdf422f" gracePeriod=30 Oct 29 23:54:16.405894 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:16.405854 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:54:17.276516 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.276460 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:54:17.365468 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.365435 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-data\") pod \"fe0b52e3-7076-420c-b899-4a8b4a3db390\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " Oct 29 23:54:17.365672 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.365488 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-token\" (UniqueName: \"kubernetes.io/projected/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-token\") pod \"fe0b52e3-7076-420c-b899-4a8b4a3db390\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " Oct 29 23:54:17.365672 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.365565 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe0b52e3-7076-420c-b899-4a8b4a3db390-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe0b52e3-7076-420c-b899-4a8b4a3db390-istiod-ca-cert\") pod \"fe0b52e3-7076-420c-b899-4a8b4a3db390\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " Oct 29 23:54:17.365672 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.365606 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-workload-socket\") pod \"fe0b52e3-7076-420c-b899-4a8b4a3db390\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " Oct 29 23:54:17.365672 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.365644 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-workload-certs\") pod \"fe0b52e3-7076-420c-b899-4a8b4a3db390\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " Oct 29 23:54:17.366003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.365678 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-envoy\") pod \"fe0b52e3-7076-420c-b899-4a8b4a3db390\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " Oct 29 23:54:17.366003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.365726 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-credential-socket\") pod \"fe0b52e3-7076-420c-b899-4a8b4a3db390\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " Oct 29 23:54:17.366003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.365745 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-data" pod "fe0b52e3-7076-420c-b899-4a8b4a3db390" (UID: "fe0b52e3-7076-420c-b899-4a8b4a3db390"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:17.366003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.365763 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe0b52e3-7076-420c-b899-4a8b4a3db390-kube-api-access-5plnf\" (UniqueName: \"kubernetes.io/projected/fe0b52e3-7076-420c-b899-4a8b4a3db390-kube-api-access-5plnf\") pod \"fe0b52e3-7076-420c-b899-4a8b4a3db390\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " Oct 29 23:54:17.366003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.365792 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-podinfo\") pod \"fe0b52e3-7076-420c-b899-4a8b4a3db390\" (UID: \"fe0b52e3-7076-420c-b899-4a8b4a3db390\") " Oct 29 23:54:17.366003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.365933 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-workload-certs" pod "fe0b52e3-7076-420c-b899-4a8b4a3db390" (UID: "fe0b52e3-7076-420c-b899-4a8b4a3db390"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:17.366382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.366084 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-credential-socket" pod "fe0b52e3-7076-420c-b899-4a8b4a3db390" (UID: "fe0b52e3-7076-420c-b899-4a8b4a3db390"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:17.366382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.366269 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0b52e3-7076-420c-b899-4a8b4a3db390-istiod-ca-cert" pod "fe0b52e3-7076-420c-b899-4a8b4a3db390" (UID: "fe0b52e3-7076-420c-b899-4a8b4a3db390"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:17.366502 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.366382 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:17.366502 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.366401 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe0b52e3-7076-420c-b899-4a8b4a3db390-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:17.366502 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.366415 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:17.366502 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.366428 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:17.366790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.366548 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-workload-socket" pod "fe0b52e3-7076-420c-b899-4a8b4a3db390" (UID: "fe0b52e3-7076-420c-b899-4a8b4a3db390"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:17.369293 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.369260 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-token" pod "fe0b52e3-7076-420c-b899-4a8b4a3db390" (UID: "fe0b52e3-7076-420c-b899-4a8b4a3db390"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:17.369506 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.369481 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-podinfo" pod "fe0b52e3-7076-420c-b899-4a8b4a3db390" (UID: "fe0b52e3-7076-420c-b899-4a8b4a3db390"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:54:17.369628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.369522 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0b52e3-7076-420c-b899-4a8b4a3db390-kube-api-access-5plnf" pod "fe0b52e3-7076-420c-b899-4a8b4a3db390" (UID: "fe0b52e3-7076-420c-b899-4a8b4a3db390"). InnerVolumeSpecName "kube-api-access-5plnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:17.369677 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.369651 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-envoy" pod "fe0b52e3-7076-420c-b899-4a8b4a3db390" (UID: "fe0b52e3-7076-420c-b899-4a8b4a3db390"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:17.467750 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.467714 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:17.467750 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.467751 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:17.468160 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.467766 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:17.468160 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.467780 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5plnf\" (UniqueName: \"kubernetes.io/projected/fe0b52e3-7076-420c-b899-4a8b4a3db390-kube-api-access-5plnf\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:17.468160 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.467794 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe0b52e3-7076-420c-b899-4a8b4a3db390-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:17.600024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.599939 5057 generic.go:356] "Generic (PLEG): container finished" podID="fe0b52e3-7076-420c-b899-4a8b4a3db390" containerID="3eedf5d6f7cb7cca0d2451364457707723a72f680ea0c729d6d43fcd3fdf422f" exitCode=0 Oct 29 23:54:17.600024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.599992 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" Oct 29 23:54:17.600024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.600008 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" event={"ID":"fe0b52e3-7076-420c-b899-4a8b4a3db390","Type":"ContainerDied","Data":"3eedf5d6f7cb7cca0d2451364457707723a72f680ea0c729d6d43fcd3fdf422f"} Oct 29 23:54:17.600311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.600053 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x" event={"ID":"fe0b52e3-7076-420c-b899-4a8b4a3db390","Type":"ContainerDied","Data":"28d42e4aadad548bfee749437e74c8227a6404dbc9a4a637ea6a8c1f29f35064"} Oct 29 23:54:17.600311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.600075 5057 scope.go:122] "RemoveContainer" containerID="3eedf5d6f7cb7cca0d2451364457707723a72f680ea0c729d6d43fcd3fdf422f" Oct 29 23:54:17.610321 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.610300 5057 scope.go:122] "RemoveContainer" containerID="d23890e7790a87be2ea6bb1cf4337883f81cf029591b064e26f77b6314ce878f" Oct 29 23:54:17.619318 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.619300 5057 scope.go:122] "RemoveContainer" containerID="3eedf5d6f7cb7cca0d2451364457707723a72f680ea0c729d6d43fcd3fdf422f" Oct 29 23:54:17.619590 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:54:17.619567 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eedf5d6f7cb7cca0d2451364457707723a72f680ea0c729d6d43fcd3fdf422f\": container with ID starting with 3eedf5d6f7cb7cca0d2451364457707723a72f680ea0c729d6d43fcd3fdf422f not found: ID does not exist" containerID="3eedf5d6f7cb7cca0d2451364457707723a72f680ea0c729d6d43fcd3fdf422f" Oct 29 23:54:17.619660 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.619599 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eedf5d6f7cb7cca0d2451364457707723a72f680ea0c729d6d43fcd3fdf422f"} err="failed to get container status \"3eedf5d6f7cb7cca0d2451364457707723a72f680ea0c729d6d43fcd3fdf422f\": rpc error: code = NotFound desc = could not find container \"3eedf5d6f7cb7cca0d2451364457707723a72f680ea0c729d6d43fcd3fdf422f\": container with ID starting with 3eedf5d6f7cb7cca0d2451364457707723a72f680ea0c729d6d43fcd3fdf422f not found: ID does not exist" Oct 29 23:54:17.619660 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.619616 5057 scope.go:122] "RemoveContainer" containerID="d23890e7790a87be2ea6bb1cf4337883f81cf029591b064e26f77b6314ce878f" Oct 29 23:54:17.619876 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:54:17.619857 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d23890e7790a87be2ea6bb1cf4337883f81cf029591b064e26f77b6314ce878f\": container with ID starting with d23890e7790a87be2ea6bb1cf4337883f81cf029591b064e26f77b6314ce878f not found: ID does not exist" containerID="d23890e7790a87be2ea6bb1cf4337883f81cf029591b064e26f77b6314ce878f" Oct 29 23:54:17.619918 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.619882 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d23890e7790a87be2ea6bb1cf4337883f81cf029591b064e26f77b6314ce878f"} err="failed to get container status \"d23890e7790a87be2ea6bb1cf4337883f81cf029591b064e26f77b6314ce878f\": rpc error: code = NotFound desc = could not find container \"d23890e7790a87be2ea6bb1cf4337883f81cf029591b064e26f77b6314ce878f\": container with ID starting with d23890e7790a87be2ea6bb1cf4337883f81cf029591b064e26f77b6314ce878f not found: ID does not exist" Oct 29 23:54:17.651949 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.651920 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x"] Oct 29 23:54:17.659009 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:17.658979 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-4e2ac7-openshift-default-56996668c7-prf4x"] Oct 29 23:54:19.482542 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:19.482502 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe0b52e3-7076-420c-b899-4a8b4a3db390" path="/var/lib/kubelet/pods/fe0b52e3-7076-420c-b899-4a8b4a3db390/volumes" Oct 29 23:54:20.343727 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.343675 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f"] Oct 29 23:54:20.343920 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.343884 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" podUID="21544662-1c4e-4386-a123-8407a2f390d4" containerName="maas-api" containerID="cri-o://298d3b3f00c8b469bff6f896bf4a92880c5d9837675c2238f2775f78b4cf2938" gracePeriod=30 Oct 29 23:54:20.617522 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.617491 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" Oct 29 23:54:20.622691 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.622655 5057 generic.go:356] "Generic (PLEG): container finished" podID="21544662-1c4e-4386-a123-8407a2f390d4" containerID="298d3b3f00c8b469bff6f896bf4a92880c5d9837675c2238f2775f78b4cf2938" exitCode=0 Oct 29 23:54:20.625287 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.622862 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" event={"ID":"21544662-1c4e-4386-a123-8407a2f390d4","Type":"ContainerDied","Data":"298d3b3f00c8b469bff6f896bf4a92880c5d9837675c2238f2775f78b4cf2938"} Oct 29 23:54:20.625287 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.622952 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" event={"ID":"21544662-1c4e-4386-a123-8407a2f390d4","Type":"ContainerDied","Data":"ffd6d1b5bb602d22d9beeef59b8d15929936073093064fbfa180b27166ceaa89"} Oct 29 23:54:20.625287 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.622985 5057 scope.go:122] "RemoveContainer" containerID="298d3b3f00c8b469bff6f896bf4a92880c5d9837675c2238f2775f78b4cf2938" Oct 29 23:54:20.625287 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.623137 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f" Oct 29 23:54:20.637602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.637580 5057 scope.go:122] "RemoveContainer" containerID="298d3b3f00c8b469bff6f896bf4a92880c5d9837675c2238f2775f78b4cf2938" Oct 29 23:54:20.637857 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:54:20.637836 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"298d3b3f00c8b469bff6f896bf4a92880c5d9837675c2238f2775f78b4cf2938\": container with ID starting with 298d3b3f00c8b469bff6f896bf4a92880c5d9837675c2238f2775f78b4cf2938 not found: ID does not exist" containerID="298d3b3f00c8b469bff6f896bf4a92880c5d9837675c2238f2775f78b4cf2938" Oct 29 23:54:20.637912 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.637866 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298d3b3f00c8b469bff6f896bf4a92880c5d9837675c2238f2775f78b4cf2938"} err="failed to get container status \"298d3b3f00c8b469bff6f896bf4a92880c5d9837675c2238f2775f78b4cf2938\": rpc error: code = NotFound desc = could not find container \"298d3b3f00c8b469bff6f896bf4a92880c5d9837675c2238f2775f78b4cf2938\": container with ID starting with 298d3b3f00c8b469bff6f896bf4a92880c5d9837675c2238f2775f78b4cf2938 not found: ID does not exist" Oct 29 23:54:20.703146 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.703111 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21544662-1c4e-4386-a123-8407a2f390d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21544662-1c4e-4386-a123-8407a2f390d4-openshift-service-ca\") pod \"21544662-1c4e-4386-a123-8407a2f390d4\" (UID: \"21544662-1c4e-4386-a123-8407a2f390d4\") " Oct 29 23:54:20.703345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.703238 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21544662-1c4e-4386-a123-8407a2f390d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21544662-1c4e-4386-a123-8407a2f390d4-metrics-certs\") pod \"21544662-1c4e-4386-a123-8407a2f390d4\" (UID: \"21544662-1c4e-4386-a123-8407a2f390d4\") " Oct 29 23:54:20.703345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.703263 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21544662-1c4e-4386-a123-8407a2f390d4-kube-api-access-4d295\" (UniqueName: \"kubernetes.io/projected/21544662-1c4e-4386-a123-8407a2f390d4-kube-api-access-4d295\") pod \"21544662-1c4e-4386-a123-8407a2f390d4\" (UID: \"21544662-1c4e-4386-a123-8407a2f390d4\") " Oct 29 23:54:20.703345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.703305 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21544662-1c4e-4386-a123-8407a2f390d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21544662-1c4e-4386-a123-8407a2f390d4-maas-api-tls\") pod \"21544662-1c4e-4386-a123-8407a2f390d4\" (UID: \"21544662-1c4e-4386-a123-8407a2f390d4\") " Oct 29 23:54:20.703530 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.703502 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21544662-1c4e-4386-a123-8407a2f390d4-openshift-service-ca" pod "21544662-1c4e-4386-a123-8407a2f390d4" (UID: "21544662-1c4e-4386-a123-8407a2f390d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:20.703665 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.703646 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21544662-1c4e-4386-a123-8407a2f390d4-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:20.705506 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.705470 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21544662-1c4e-4386-a123-8407a2f390d4-kube-api-access-4d295" pod "21544662-1c4e-4386-a123-8407a2f390d4" (UID: "21544662-1c4e-4386-a123-8407a2f390d4"). InnerVolumeSpecName "kube-api-access-4d295". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:20.705634 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.705566 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21544662-1c4e-4386-a123-8407a2f390d4-metrics-certs" pod "21544662-1c4e-4386-a123-8407a2f390d4" (UID: "21544662-1c4e-4386-a123-8407a2f390d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:54:20.705700 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.705673 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21544662-1c4e-4386-a123-8407a2f390d4-maas-api-tls" pod "21544662-1c4e-4386-a123-8407a2f390d4" (UID: "21544662-1c4e-4386-a123-8407a2f390d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:54:20.805273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.805231 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21544662-1c4e-4386-a123-8407a2f390d4-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:20.805273 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.805275 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d295\" (UniqueName: \"kubernetes.io/projected/21544662-1c4e-4386-a123-8407a2f390d4-kube-api-access-4d295\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:20.805548 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.805291 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21544662-1c4e-4386-a123-8407a2f390d4-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:20.970437 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.970400 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f"] Oct 29 23:54:20.972735 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:20.972707 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2126fb-7b648c9558-xck5f"] Oct 29 23:54:21.480393 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:21.480356 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21544662-1c4e-4386-a123-8407a2f390d4" path="/var/lib/kubelet/pods/21544662-1c4e-4386-a123-8407a2f390d4/volumes" Oct 29 23:54:21.973514 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:21.973468 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2" podUID="e3a0a5b1-c514-4685-a97d-c75ea6882e82" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:54:21.973899 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:21.973542 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr" podUID="87936d4c-796b-4176-b54a-4b520e0b7ce5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:54:23.291462 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:23.291422 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8"] Oct 29 23:54:23.325586 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:23.325547 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4"] Oct 29 23:54:26.405397 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405362 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb"] Oct 29 23:54:26.405831 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405818 5057 cpu_manager.go:414] "containerMap: removing container" podUID="44008edf-7b81-4504-adb7-132388744c6e" containerName="istio-proxy" Oct 29 23:54:26.405874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405831 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44008edf-7b81-4504-adb7-132388744c6e" containerName="istio-proxy" Oct 29 23:54:26.405874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405843 5057 cpu_manager.go:414] "containerMap: removing container" podUID="fe0b52e3-7076-420c-b899-4a8b4a3db390" containerName="istio-proxy" Oct 29 23:54:26.405874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405849 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe0b52e3-7076-420c-b899-4a8b4a3db390" containerName="istio-proxy" Oct 29 23:54:26.405874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405856 5057 cpu_manager.go:414] "containerMap: removing container" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" Oct 29 23:54:26.405874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405862 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" Oct 29 23:54:26.405874 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405873 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f99eb02d-30df-4333-b4a3-ea8b5ff576cf" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405878 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f99eb02d-30df-4333-b4a3-ea8b5ff576cf" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405885 5057 cpu_manager.go:414] "containerMap: removing container" podUID="21544662-1c4e-4386-a123-8407a2f390d4" containerName="maas-api" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405890 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21544662-1c4e-4386-a123-8407a2f390d4" containerName="maas-api" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405896 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a28590ff-f920-4993-b5f2-d8afb67b3685" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405901 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a28590ff-f920-4993-b5f2-d8afb67b3685" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405909 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f523466e-3408-48cc-aa67-21eec54c0eb0" containerName="maas-api" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405913 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f523466e-3408-48cc-aa67-21eec54c0eb0" containerName="maas-api" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405921 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a28590ff-f920-4993-b5f2-d8afb67b3685" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405926 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a28590ff-f920-4993-b5f2-d8afb67b3685" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405933 5057 cpu_manager.go:414] "containerMap: removing container" podUID="804a9cae-91df-441a-a59d-e8c117344b78" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405937 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="804a9cae-91df-441a-a59d-e8c117344b78" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405944 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f99eb02d-30df-4333-b4a3-ea8b5ff576cf" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405948 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f99eb02d-30df-4333-b4a3-ea8b5ff576cf" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405953 5057 cpu_manager.go:414] "containerMap: removing container" podUID="44008edf-7b81-4504-adb7-132388744c6e" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405958 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44008edf-7b81-4504-adb7-132388744c6e" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405965 5057 cpu_manager.go:414] "containerMap: removing container" podUID="c6508b22-be11-4d60-80bd-42a88a299853" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405970 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6508b22-be11-4d60-80bd-42a88a299853" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405976 5057 cpu_manager.go:414] "containerMap: removing container" podUID="52937560-4419-4c5a-b4a1-1918d3963e22" containerName="maas-api" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405980 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52937560-4419-4c5a-b4a1-1918d3963e22" containerName="maas-api" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405987 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f255ae1a-010b-4f50-9414-5b09b507c085" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405991 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f255ae1a-010b-4f50-9414-5b09b507c085" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.405997 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e5ee3546-b9fa-49f9-a7b1-c110932ebb10" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406002 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ee3546-b9fa-49f9-a7b1-c110932ebb10" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406013 5057 cpu_manager.go:414] "containerMap: removing container" podUID="41da29c2-0216-4adf-88ba-32d655e88efd" containerName="payload-pre-processing" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406020 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41da29c2-0216-4adf-88ba-32d655e88efd" containerName="payload-pre-processing" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406027 5057 cpu_manager.go:414] "containerMap: removing container" podUID="c6508b22-be11-4d60-80bd-42a88a299853" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406031 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6508b22-be11-4d60-80bd-42a88a299853" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406040 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406044 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406051 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4104e017-b65b-404f-8226-71df8ca5c903" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406055 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4104e017-b65b-404f-8226-71df8ca5c903" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406065 5057 cpu_manager.go:414] "containerMap: removing container" podUID="c9adfb1e-b9a5-4f30-9bba-ef91a855ff71" containerName="payload-pre-processing" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406071 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9adfb1e-b9a5-4f30-9bba-ef91a855ff71" containerName="payload-pre-processing" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406081 5057 cpu_manager.go:414] "containerMap: removing container" podUID="98609461-cc7d-449e-8dc9-f035a5c53d94" containerName="payload-processing" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406085 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98609461-cc7d-449e-8dc9-f035a5c53d94" containerName="payload-processing" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406092 5057 cpu_manager.go:414] "containerMap: removing container" podUID="87c126a1-a9e4-4aa9-afe9-78727290d709" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406096 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c126a1-a9e4-4aa9-afe9-78727290d709" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406103 5057 cpu_manager.go:414] "containerMap: removing container" podUID="23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1" containerName="maas-api" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406107 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1" containerName="maas-api" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406114 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4104e017-b65b-404f-8226-71df8ca5c903" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406118 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4104e017-b65b-404f-8226-71df8ca5c903" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406125 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d032ce48-7955-47e5-a74b-9af1ef81ed17" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406130 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d032ce48-7955-47e5-a74b-9af1ef81ed17" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406136 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5c575af2-2a8a-4e83-adcc-7d6578c53235" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406140 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c575af2-2a8a-4e83-adcc-7d6578c53235" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406147 5057 cpu_manager.go:414] "containerMap: removing container" podUID="153fed7b-8844-4d52-a5ad-0b6d8eda52e7" containerName="payload-processing" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406151 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153fed7b-8844-4d52-a5ad-0b6d8eda52e7" containerName="payload-processing" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406157 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5c575af2-2a8a-4e83-adcc-7d6578c53235" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406163 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c575af2-2a8a-4e83-adcc-7d6578c53235" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406170 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1de79fc1-b30d-40eb-9f6d-55b1ddde3139" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406175 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1de79fc1-b30d-40eb-9f6d-55b1ddde3139" containerName="istio-proxy" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406198 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4ecbde51-118e-44b3-ae0c-491e4d03f124" containerName="maas-api" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406202 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ecbde51-118e-44b3-ae0c-491e4d03f124" containerName="maas-api" Oct 29 23:54:26.406153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406208 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a8aa349b-e04a-4726-9b2e-a910d328e7f9" containerName="payload-pre-processing" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406213 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8aa349b-e04a-4726-9b2e-a910d328e7f9" containerName="payload-pre-processing" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406222 5057 cpu_manager.go:414] "containerMap: removing container" podUID="ed4c81ea-a9b7-43c6-a893-342cd3d4389f" containerName="payload-processing" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406227 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed4c81ea-a9b7-43c6-a893-342cd3d4389f" containerName="payload-processing" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406234 5057 cpu_manager.go:414] "containerMap: removing container" podUID="cb34f658-47bb-4941-bdca-53d80e885bf6" containerName="payload-pre-processing" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406239 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb34f658-47bb-4941-bdca-53d80e885bf6" containerName="payload-pre-processing" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406248 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4494b17b-7e7b-41e4-a759-0284b87f5a57" containerName="payload-pre-processing" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406253 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4494b17b-7e7b-41e4-a759-0284b87f5a57" containerName="payload-pre-processing" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406259 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1ae60c2b-2e14-416e-b552-fb9b929be559" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406265 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae60c2b-2e14-416e-b552-fb9b929be559" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406274 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b" containerName="payload-pre-processing" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406278 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b" containerName="payload-pre-processing" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406289 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9b1937f5-1a2b-4a0d-a42a-636c5b6aafca" containerName="maas-api" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406293 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b1937f5-1a2b-4a0d-a42a-636c5b6aafca" containerName="maas-api" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406300 5057 cpu_manager.go:414] "containerMap: removing container" podUID="abfee661-d5d7-4b6f-8691-aec67412f360" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406305 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abfee661-d5d7-4b6f-8691-aec67412f360" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406386 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe0b52e3-7076-420c-b899-4a8b4a3db390" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406395 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d032ce48-7955-47e5-a74b-9af1ef81ed17" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406402 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="abfee661-d5d7-4b6f-8691-aec67412f360" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406409 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="44008edf-7b81-4504-adb7-132388744c6e" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406414 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe0b52e3-7076-420c-b899-4a8b4a3db390" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406421 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="017ea90c-95e4-4199-8f90-e67ee5f80c41" containerName="main" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406427 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="21544662-1c4e-4386-a123-8407a2f390d4" containerName="maas-api" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406433 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f99eb02d-30df-4333-b4a3-ea8b5ff576cf" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406439 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a28590ff-f920-4993-b5f2-d8afb67b3685" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406444 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f523466e-3408-48cc-aa67-21eec54c0eb0" containerName="maas-api" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406448 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a28590ff-f920-4993-b5f2-d8afb67b3685" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406454 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="804a9cae-91df-441a-a59d-e8c117344b78" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406460 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f99eb02d-30df-4333-b4a3-ea8b5ff576cf" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406464 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="44008edf-7b81-4504-adb7-132388744c6e" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406469 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6508b22-be11-4d60-80bd-42a88a299853" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406473 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="52937560-4419-4c5a-b4a1-1918d3963e22" containerName="maas-api" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406478 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f255ae1a-010b-4f50-9414-5b09b507c085" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406485 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ee3546-b9fa-49f9-a7b1-c110932ebb10" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406494 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="41da29c2-0216-4adf-88ba-32d655e88efd" containerName="payload-pre-processing" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406501 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6508b22-be11-4d60-80bd-42a88a299853" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406510 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a56baba-0899-4946-89d0-13db7c5c0242" containerName="main" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406519 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4104e017-b65b-404f-8226-71df8ca5c903" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406528 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9adfb1e-b9a5-4f30-9bba-ef91a855ff71" containerName="payload-pre-processing" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406537 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="98609461-cc7d-449e-8dc9-f035a5c53d94" containerName="payload-processing" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406544 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c126a1-a9e4-4aa9-afe9-78727290d709" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406552 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d06bfc-9f53-4fe8-85a0-f2721b2a7cf1" containerName="maas-api" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406568 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4104e017-b65b-404f-8226-71df8ca5c903" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406578 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c575af2-2a8a-4e83-adcc-7d6578c53235" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406586 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="153fed7b-8844-4d52-a5ad-0b6d8eda52e7" containerName="payload-processing" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406593 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c575af2-2a8a-4e83-adcc-7d6578c53235" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406601 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1de79fc1-b30d-40eb-9f6d-55b1ddde3139" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406609 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ecbde51-118e-44b3-ae0c-491e4d03f124" containerName="maas-api" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406619 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8aa349b-e04a-4726-9b2e-a910d328e7f9" containerName="payload-pre-processing" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406628 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed4c81ea-a9b7-43c6-a893-342cd3d4389f" containerName="payload-processing" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406637 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb34f658-47bb-4941-bdca-53d80e885bf6" containerName="payload-pre-processing" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406646 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4494b17b-7e7b-41e4-a759-0284b87f5a57" containerName="payload-pre-processing" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406653 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae60c2b-2e14-416e-b552-fb9b929be559" containerName="istio-proxy" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406662 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f7ac761-6de5-4d8a-97a5-f8f0e5cbfc9b" containerName="payload-pre-processing" Oct 29 23:54:26.408575 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.406673 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b1937f5-1a2b-4a0d-a42a-636c5b6aafca" containerName="maas-api" Oct 29 23:54:26.411650 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.411631 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb" Oct 29 23:54:26.436251 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.432730 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb"] Oct 29 23:54:26.506355 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.506278 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:54:26.506567 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.506379 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:54:26.574259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.574220 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjxvp\" (UniqueName: \"kubernetes.io/projected/c82f8e25-d512-4f0d-9148-5ae241c0424b-kube-api-access-cjxvp\") pod \"maas-api-revoke-sub-df20bf02-fzvcb\" (UID: \"c82f8e25-d512-4f0d-9148-5ae241c0424b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb" Oct 29 23:54:26.574474 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.574269 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82f8e25-d512-4f0d-9148-5ae241c0424b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df20bf02-fzvcb\" (UID: \"c82f8e25-d512-4f0d-9148-5ae241c0424b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb" Oct 29 23:54:26.574474 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.574292 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82f8e25-d512-4f0d-9148-5ae241c0424b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df20bf02-fzvcb\" (UID: \"c82f8e25-d512-4f0d-9148-5ae241c0424b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb" Oct 29 23:54:26.675657 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.675614 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82f8e25-d512-4f0d-9148-5ae241c0424b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df20bf02-fzvcb\" (UID: \"c82f8e25-d512-4f0d-9148-5ae241c0424b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb" Oct 29 23:54:26.675887 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.675761 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjxvp\" (UniqueName: \"kubernetes.io/projected/c82f8e25-d512-4f0d-9148-5ae241c0424b-kube-api-access-cjxvp\") pod \"maas-api-revoke-sub-df20bf02-fzvcb\" (UID: \"c82f8e25-d512-4f0d-9148-5ae241c0424b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb" Oct 29 23:54:26.675887 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.675805 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82f8e25-d512-4f0d-9148-5ae241c0424b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df20bf02-fzvcb\" (UID: \"c82f8e25-d512-4f0d-9148-5ae241c0424b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb" Oct 29 23:54:26.676533 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.676506 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82f8e25-d512-4f0d-9148-5ae241c0424b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df20bf02-fzvcb\" (UID: \"c82f8e25-d512-4f0d-9148-5ae241c0424b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb" Oct 29 23:54:26.678003 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.677978 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82f8e25-d512-4f0d-9148-5ae241c0424b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df20bf02-fzvcb\" (UID: \"c82f8e25-d512-4f0d-9148-5ae241c0424b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb" Oct 29 23:54:26.706535 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.706501 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjxvp\" (UniqueName: \"kubernetes.io/projected/c82f8e25-d512-4f0d-9148-5ae241c0424b-kube-api-access-cjxvp\") pod \"maas-api-revoke-sub-df20bf02-fzvcb\" (UID: \"c82f8e25-d512-4f0d-9148-5ae241c0424b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb" Oct 29 23:54:26.726518 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:26.726487 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb" Oct 29 23:54:27.036178 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:27.036148 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb"] Oct 29 23:54:27.040026 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:54:27.039990 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82f8e25_d512_4f0d_9148_5ae241c0424b.slice/crio-f0ec7d5ccacb1923b056a69eb27a98163db67970052963277f4cf5e3996654f5 WatchSource:0}: Error finding container f0ec7d5ccacb1923b056a69eb27a98163db67970052963277f4cf5e3996654f5: Status 404 returned error can't find the container with id f0ec7d5ccacb1923b056a69eb27a98163db67970052963277f4cf5e3996654f5 Oct 29 23:54:27.666921 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:27.666814 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb" event={"ID":"c82f8e25-d512-4f0d-9148-5ae241c0424b","Type":"ContainerStarted","Data":"0f11532eff46873c329fd042c03240a7833ac2d218d4d68684ded69ed8828c65"} Oct 29 23:54:27.666921 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:27.666889 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb" event={"ID":"c82f8e25-d512-4f0d-9148-5ae241c0424b","Type":"ContainerStarted","Data":"f0ec7d5ccacb1923b056a69eb27a98163db67970052963277f4cf5e3996654f5"} Oct 29 23:54:27.709506 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:27.709452 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb" podStartSLOduration=1.709438702 podStartE2EDuration="1.709438702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:54:27.708788926 +0000 UTC m=+4314.360287456" watchObservedRunningTime="2026-10-29 23:54:27.709438702 +0000 UTC m=+4314.360937232" Oct 29 23:54:28.291836 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.291788 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8" podUID="6b25834c-7d18-46f7-85ea-c536e5eb3ea0" containerName="payload-processing" containerID="cri-o://3ea3cbe3dfeab5910d6a99e2ecb14c932783eae7212599dcd0df27f1658368db" gracePeriod=25 Oct 29 23:54:28.315620 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.315586 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl"] Oct 29 23:54:28.315849 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.315799 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" podUID="e35e101c-582d-4121-9792-0f02cc8c528e" containerName="maas-api" containerID="cri-o://5e5733fcbe119f82d84769cf2b591ecb1cfcd2b0cfe94a71fc26b512758ca795" gracePeriod=30 Oct 29 23:54:28.327018 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.326747 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4" podUID="9655e209-6b99-40ea-9602-51eceac7efa9" containerName="payload-pre-processing" containerID="cri-o://8b2228c1e639b8054be991efcca4d2d95c608ffc068e1ccfc47aa8664cf6b69d" gracePeriod=25 Oct 29 23:54:28.591149 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.591124 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" Oct 29 23:54:28.595341 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.595315 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e35e101c-582d-4121-9792-0f02cc8c528e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e35e101c-582d-4121-9792-0f02cc8c528e-metrics-certs\") pod \"e35e101c-582d-4121-9792-0f02cc8c528e\" (UID: \"e35e101c-582d-4121-9792-0f02cc8c528e\") " Oct 29 23:54:28.595508 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.595376 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e35e101c-582d-4121-9792-0f02cc8c528e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e35e101c-582d-4121-9792-0f02cc8c528e-openshift-service-ca\") pod \"e35e101c-582d-4121-9792-0f02cc8c528e\" (UID: \"e35e101c-582d-4121-9792-0f02cc8c528e\") " Oct 29 23:54:28.595508 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.595477 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e35e101c-582d-4121-9792-0f02cc8c528e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e35e101c-582d-4121-9792-0f02cc8c528e-maas-api-tls\") pod \"e35e101c-582d-4121-9792-0f02cc8c528e\" (UID: \"e35e101c-582d-4121-9792-0f02cc8c528e\") " Oct 29 23:54:28.595640 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.595596 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e35e101c-582d-4121-9792-0f02cc8c528e-kube-api-access-vg6ww\" (UniqueName: \"kubernetes.io/projected/e35e101c-582d-4121-9792-0f02cc8c528e-kube-api-access-vg6ww\") pod \"e35e101c-582d-4121-9792-0f02cc8c528e\" (UID: \"e35e101c-582d-4121-9792-0f02cc8c528e\") " Oct 29 23:54:28.595742 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.595722 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e35e101c-582d-4121-9792-0f02cc8c528e-openshift-service-ca" pod "e35e101c-582d-4121-9792-0f02cc8c528e" (UID: "e35e101c-582d-4121-9792-0f02cc8c528e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:28.596039 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.596016 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e35e101c-582d-4121-9792-0f02cc8c528e-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:28.597736 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.597716 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e35e101c-582d-4121-9792-0f02cc8c528e-metrics-certs" pod "e35e101c-582d-4121-9792-0f02cc8c528e" (UID: "e35e101c-582d-4121-9792-0f02cc8c528e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:54:28.597975 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.597954 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e35e101c-582d-4121-9792-0f02cc8c528e-maas-api-tls" pod "e35e101c-582d-4121-9792-0f02cc8c528e" (UID: "e35e101c-582d-4121-9792-0f02cc8c528e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:54:28.598156 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.598135 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e35e101c-582d-4121-9792-0f02cc8c528e-kube-api-access-vg6ww" pod "e35e101c-582d-4121-9792-0f02cc8c528e" (UID: "e35e101c-582d-4121-9792-0f02cc8c528e"). InnerVolumeSpecName "kube-api-access-vg6ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:28.674784 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.674741 5057 generic.go:356] "Generic (PLEG): container finished" podID="c82f8e25-d512-4f0d-9148-5ae241c0424b" containerID="0f11532eff46873c329fd042c03240a7833ac2d218d4d68684ded69ed8828c65" exitCode=0 Oct 29 23:54:28.675276 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.674849 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb" event={"ID":"c82f8e25-d512-4f0d-9148-5ae241c0424b","Type":"ContainerDied","Data":"0f11532eff46873c329fd042c03240a7833ac2d218d4d68684ded69ed8828c65"} Oct 29 23:54:28.676169 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.676145 5057 generic.go:356] "Generic (PLEG): container finished" podID="e35e101c-582d-4121-9792-0f02cc8c528e" containerID="5e5733fcbe119f82d84769cf2b591ecb1cfcd2b0cfe94a71fc26b512758ca795" exitCode=0 Oct 29 23:54:28.676307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.676212 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" Oct 29 23:54:28.676307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.676234 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" event={"ID":"e35e101c-582d-4121-9792-0f02cc8c528e","Type":"ContainerDied","Data":"5e5733fcbe119f82d84769cf2b591ecb1cfcd2b0cfe94a71fc26b512758ca795"} Oct 29 23:54:28.676307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.676282 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl" event={"ID":"e35e101c-582d-4121-9792-0f02cc8c528e","Type":"ContainerDied","Data":"9044fe043a3a1c367532bbc19110341643566449bb78fd9ec6c6119f80bd1a6a"} Oct 29 23:54:28.676307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.676304 5057 scope.go:122] "RemoveContainer" containerID="5e5733fcbe119f82d84769cf2b591ecb1cfcd2b0cfe94a71fc26b512758ca795" Oct 29 23:54:28.695853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.695832 5057 scope.go:122] "RemoveContainer" containerID="5e5733fcbe119f82d84769cf2b591ecb1cfcd2b0cfe94a71fc26b512758ca795" Oct 29 23:54:28.696252 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:54:28.696219 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e5733fcbe119f82d84769cf2b591ecb1cfcd2b0cfe94a71fc26b512758ca795\": container with ID starting with 5e5733fcbe119f82d84769cf2b591ecb1cfcd2b0cfe94a71fc26b512758ca795 not found: ID does not exist" containerID="5e5733fcbe119f82d84769cf2b591ecb1cfcd2b0cfe94a71fc26b512758ca795" Oct 29 23:54:28.696347 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.696267 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e5733fcbe119f82d84769cf2b591ecb1cfcd2b0cfe94a71fc26b512758ca795"} err="failed to get container status \"5e5733fcbe119f82d84769cf2b591ecb1cfcd2b0cfe94a71fc26b512758ca795\": rpc error: code = NotFound desc = could not find container \"5e5733fcbe119f82d84769cf2b591ecb1cfcd2b0cfe94a71fc26b512758ca795\": container with ID starting with 5e5733fcbe119f82d84769cf2b591ecb1cfcd2b0cfe94a71fc26b512758ca795 not found: ID does not exist" Oct 29 23:54:28.696606 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.696588 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e35e101c-582d-4121-9792-0f02cc8c528e-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:28.696700 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.696612 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e35e101c-582d-4121-9792-0f02cc8c528e-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:28.696700 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.696626 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg6ww\" (UniqueName: \"kubernetes.io/projected/e35e101c-582d-4121-9792-0f02cc8c528e-kube-api-access-vg6ww\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:28.753598 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.753547 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl"] Oct 29 23:54:28.755935 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:28.755910 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2077b4-84dcbb7768-p8gpl"] Oct 29 23:54:29.484816 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:29.484778 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e35e101c-582d-4121-9792-0f02cc8c528e" path="/var/lib/kubelet/pods/e35e101c-582d-4121-9792-0f02cc8c528e/volumes" Oct 29 23:54:29.841963 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:29.841929 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb" Oct 29 23:54:29.911124 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:29.911087 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c82f8e25-d512-4f0d-9148-5ae241c0424b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82f8e25-d512-4f0d-9148-5ae241c0424b-maas-api-xks-ca\") pod \"c82f8e25-d512-4f0d-9148-5ae241c0424b\" (UID: \"c82f8e25-d512-4f0d-9148-5ae241c0424b\") " Oct 29 23:54:29.911417 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:29.911136 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c82f8e25-d512-4f0d-9148-5ae241c0424b-kube-api-access-cjxvp\" (UniqueName: \"kubernetes.io/projected/c82f8e25-d512-4f0d-9148-5ae241c0424b-kube-api-access-cjxvp\") pod \"c82f8e25-d512-4f0d-9148-5ae241c0424b\" (UID: \"c82f8e25-d512-4f0d-9148-5ae241c0424b\") " Oct 29 23:54:29.911417 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:29.911165 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c82f8e25-d512-4f0d-9148-5ae241c0424b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82f8e25-d512-4f0d-9148-5ae241c0424b-maas-api-openshift-ca\") pod \"c82f8e25-d512-4f0d-9148-5ae241c0424b\" (UID: \"c82f8e25-d512-4f0d-9148-5ae241c0424b\") " Oct 29 23:54:29.911646 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:29.911621 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82f8e25-d512-4f0d-9148-5ae241c0424b-maas-api-openshift-ca" pod "c82f8e25-d512-4f0d-9148-5ae241c0424b" (UID: "c82f8e25-d512-4f0d-9148-5ae241c0424b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:29.913780 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:29.913750 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82f8e25-d512-4f0d-9148-5ae241c0424b-kube-api-access-cjxvp" pod "c82f8e25-d512-4f0d-9148-5ae241c0424b" (UID: "c82f8e25-d512-4f0d-9148-5ae241c0424b"). InnerVolumeSpecName "kube-api-access-cjxvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:29.914366 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:29.914337 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c82f8e25-d512-4f0d-9148-5ae241c0424b-maas-api-xks-ca" pod "c82f8e25-d512-4f0d-9148-5ae241c0424b" (UID: "c82f8e25-d512-4f0d-9148-5ae241c0424b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:54:30.013311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:30.013225 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82f8e25-d512-4f0d-9148-5ae241c0424b-maas-api-xks-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:30.013311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:30.013263 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjxvp\" (UniqueName: \"kubernetes.io/projected/c82f8e25-d512-4f0d-9148-5ae241c0424b-kube-api-access-cjxvp\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:30.013311 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:30.013279 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82f8e25-d512-4f0d-9148-5ae241c0424b-maas-api-openshift-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:30.690672 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:30.690638 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb" Oct 29 23:54:30.690872 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:30.690667 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df20bf02-fzvcb" event={"ID":"c82f8e25-d512-4f0d-9148-5ae241c0424b","Type":"ContainerDied","Data":"f0ec7d5ccacb1923b056a69eb27a98163db67970052963277f4cf5e3996654f5"} Oct 29 23:54:30.690872 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:30.690722 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0ec7d5ccacb1923b056a69eb27a98163db67970052963277f4cf5e3996654f5" Oct 29 23:54:31.476721 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.476693 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr" Oct 29 23:54:31.517136 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.517108 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2" Oct 29 23:54:31.530888 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.530854 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87936d4c-796b-4176-b54a-4b520e0b7ce5-kube-api-access-mt9wn\" (UniqueName: \"kubernetes.io/projected/87936d4c-796b-4176-b54a-4b520e0b7ce5-kube-api-access-mt9wn\") pod \"87936d4c-796b-4176-b54a-4b520e0b7ce5\" (UID: \"87936d4c-796b-4176-b54a-4b520e0b7ce5\") " Oct 29 23:54:31.531046 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.530957 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87936d4c-796b-4176-b54a-4b520e0b7ce5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87936d4c-796b-4176-b54a-4b520e0b7ce5-plugins-config-volume\") pod \"87936d4c-796b-4176-b54a-4b520e0b7ce5\" (UID: \"87936d4c-796b-4176-b54a-4b520e0b7ce5\") " Oct 29 23:54:31.531091 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.531059 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3a0a5b1-c514-4685-a97d-c75ea6882e82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a0a5b1-c514-4685-a97d-c75ea6882e82-plugins-config-volume\") pod \"e3a0a5b1-c514-4685-a97d-c75ea6882e82\" (UID: \"e3a0a5b1-c514-4685-a97d-c75ea6882e82\") " Oct 29 23:54:31.531157 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.531100 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3a0a5b1-c514-4685-a97d-c75ea6882e82-kube-api-access-d7cvp\" (UniqueName: \"kubernetes.io/projected/e3a0a5b1-c514-4685-a97d-c75ea6882e82-kube-api-access-d7cvp\") pod \"e3a0a5b1-c514-4685-a97d-c75ea6882e82\" (UID: \"e3a0a5b1-c514-4685-a97d-c75ea6882e82\") " Oct 29 23:54:31.532103 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.532073 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87936d4c-796b-4176-b54a-4b520e0b7ce5-plugins-config-volume" pod "87936d4c-796b-4176-b54a-4b520e0b7ce5" (UID: "87936d4c-796b-4176-b54a-4b520e0b7ce5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:31.532304 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.532174 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3a0a5b1-c514-4685-a97d-c75ea6882e82-plugins-config-volume" pod "e3a0a5b1-c514-4685-a97d-c75ea6882e82" (UID: "e3a0a5b1-c514-4685-a97d-c75ea6882e82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:31.533853 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.533820 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87936d4c-796b-4176-b54a-4b520e0b7ce5-kube-api-access-mt9wn" pod "87936d4c-796b-4176-b54a-4b520e0b7ce5" (UID: "87936d4c-796b-4176-b54a-4b520e0b7ce5"). InnerVolumeSpecName "kube-api-access-mt9wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:31.533977 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.533936 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a0a5b1-c514-4685-a97d-c75ea6882e82-kube-api-access-d7cvp" pod "e3a0a5b1-c514-4685-a97d-c75ea6882e82" (UID: "e3a0a5b1-c514-4685-a97d-c75ea6882e82"). InnerVolumeSpecName "kube-api-access-d7cvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:31.587798 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.587752 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7"] Oct 29 23:54:31.588522 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.588496 5057 cpu_manager.go:414] "containerMap: removing container" podUID="c82f8e25-d512-4f0d-9148-5ae241c0424b" containerName="revoke-keys" Oct 29 23:54:31.588522 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.588520 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c82f8e25-d512-4f0d-9148-5ae241c0424b" containerName="revoke-keys" Oct 29 23:54:31.588744 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.588545 5057 cpu_manager.go:414] "containerMap: removing container" podUID="87936d4c-796b-4176-b54a-4b520e0b7ce5" containerName="payload-processing" Oct 29 23:54:31.588744 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.588553 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87936d4c-796b-4176-b54a-4b520e0b7ce5" containerName="payload-processing" Oct 29 23:54:31.588744 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.588567 5057 cpu_manager.go:414] "containerMap: removing container" podUID="fe0b52e3-7076-420c-b899-4a8b4a3db390" containerName="istio-proxy" Oct 29 23:54:31.588744 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.588575 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe0b52e3-7076-420c-b899-4a8b4a3db390" containerName="istio-proxy" Oct 29 23:54:31.588744 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.588589 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e3a0a5b1-c514-4685-a97d-c75ea6882e82" containerName="payload-pre-processing" Oct 29 23:54:31.588744 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.588596 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3a0a5b1-c514-4685-a97d-c75ea6882e82" containerName="payload-pre-processing" Oct 29 23:54:31.588744 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.588614 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e35e101c-582d-4121-9792-0f02cc8c528e" containerName="maas-api" Oct 29 23:54:31.588744 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.588621 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e35e101c-582d-4121-9792-0f02cc8c528e" containerName="maas-api" Oct 29 23:54:31.588744 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.588633 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f255ae1a-010b-4f50-9414-5b09b507c085" containerName="istio-proxy" Oct 29 23:54:31.588744 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.588640 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f255ae1a-010b-4f50-9414-5b09b507c085" containerName="istio-proxy" Oct 29 23:54:31.588744 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.588662 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d032ce48-7955-47e5-a74b-9af1ef81ed17" containerName="istio-proxy" Oct 29 23:54:31.588744 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.588669 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d032ce48-7955-47e5-a74b-9af1ef81ed17" containerName="istio-proxy" Oct 29 23:54:31.589440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.588802 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="c82f8e25-d512-4f0d-9148-5ae241c0424b" containerName="revoke-keys" Oct 29 23:54:31.589440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.588814 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d032ce48-7955-47e5-a74b-9af1ef81ed17" containerName="istio-proxy" Oct 29 23:54:31.589440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.588825 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="87936d4c-796b-4176-b54a-4b520e0b7ce5" containerName="payload-processing" Oct 29 23:54:31.589440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.588836 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3a0a5b1-c514-4685-a97d-c75ea6882e82" containerName="payload-pre-processing" Oct 29 23:54:31.589440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.588847 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e35e101c-582d-4121-9792-0f02cc8c528e" containerName="maas-api" Oct 29 23:54:31.589440 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.588856 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f255ae1a-010b-4f50-9414-5b09b507c085" containerName="istio-proxy" Oct 29 23:54:31.592760 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.592737 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7" Oct 29 23:54:31.612847 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.612815 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7"] Oct 29 23:54:31.633081 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.632992 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29396ecc-dbb0-4fca-a886-99496af5bbd2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7\" (UID: \"29396ecc-dbb0-4fca-a886-99496af5bbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7" Oct 29 23:54:31.633760 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.633513 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29396ecc-dbb0-4fca-a886-99496af5bbd2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7\" (UID: \"29396ecc-dbb0-4fca-a886-99496af5bbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7" Oct 29 23:54:31.633760 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.633558 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b9kv\" (UniqueName: \"kubernetes.io/projected/29396ecc-dbb0-4fca-a886-99496af5bbd2-kube-api-access-6b9kv\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7\" (UID: \"29396ecc-dbb0-4fca-a886-99496af5bbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7" Oct 29 23:54:31.633760 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.633664 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a0a5b1-c514-4685-a97d-c75ea6882e82-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:31.633760 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.633680 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7cvp\" (UniqueName: \"kubernetes.io/projected/e3a0a5b1-c514-4685-a97d-c75ea6882e82-kube-api-access-d7cvp\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:31.633760 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.633696 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt9wn\" (UniqueName: \"kubernetes.io/projected/87936d4c-796b-4176-b54a-4b520e0b7ce5-kube-api-access-mt9wn\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:31.633760 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.633710 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87936d4c-796b-4176-b54a-4b520e0b7ce5-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:31.658772 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.658736 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:54:31.658937 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.658797 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:54:31.699031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.698940 5057 generic.go:356] "Generic (PLEG): container finished" podID="87936d4c-796b-4176-b54a-4b520e0b7ce5" containerID="63afa5a3a93d6f74b869d00e48fb75c41899dd8f639a9117fe3cff87b35472ce" exitCode=0 Oct 29 23:54:31.699031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.698985 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr" Oct 29 23:54:31.699031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.699016 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr" event={"ID":"87936d4c-796b-4176-b54a-4b520e0b7ce5","Type":"ContainerDied","Data":"63afa5a3a93d6f74b869d00e48fb75c41899dd8f639a9117fe3cff87b35472ce"} Oct 29 23:54:31.699351 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.699065 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr" event={"ID":"87936d4c-796b-4176-b54a-4b520e0b7ce5","Type":"ContainerDied","Data":"793a122a3fcbe5115e812b0d39451ba3c72271a7da8b23af282be0805489f178"} Oct 29 23:54:31.699351 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.699095 5057 scope.go:122] "RemoveContainer" containerID="63afa5a3a93d6f74b869d00e48fb75c41899dd8f639a9117fe3cff87b35472ce" Oct 29 23:54:31.700607 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.700583 5057 generic.go:356] "Generic (PLEG): container finished" podID="e3a0a5b1-c514-4685-a97d-c75ea6882e82" containerID="c4ce3d9796ce9730a641d772c9bd023d34ec70de5091cb8b3945920278671b4f" exitCode=0 Oct 29 23:54:31.700738 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.700614 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2" event={"ID":"e3a0a5b1-c514-4685-a97d-c75ea6882e82","Type":"ContainerDied","Data":"c4ce3d9796ce9730a641d772c9bd023d34ec70de5091cb8b3945920278671b4f"} Oct 29 23:54:31.700738 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.700634 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2" Oct 29 23:54:31.700738 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.700666 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2" event={"ID":"e3a0a5b1-c514-4685-a97d-c75ea6882e82","Type":"ContainerDied","Data":"6cee3260932de976e4d63b0c03088cc43961952e7e91d0a87ccd0ba43622b174"} Oct 29 23:54:31.710350 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.710325 5057 scope.go:122] "RemoveContainer" containerID="63afa5a3a93d6f74b869d00e48fb75c41899dd8f639a9117fe3cff87b35472ce" Oct 29 23:54:31.710719 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:54:31.710695 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63afa5a3a93d6f74b869d00e48fb75c41899dd8f639a9117fe3cff87b35472ce\": container with ID starting with 63afa5a3a93d6f74b869d00e48fb75c41899dd8f639a9117fe3cff87b35472ce not found: ID does not exist" containerID="63afa5a3a93d6f74b869d00e48fb75c41899dd8f639a9117fe3cff87b35472ce" Oct 29 23:54:31.710792 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.710731 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63afa5a3a93d6f74b869d00e48fb75c41899dd8f639a9117fe3cff87b35472ce"} err="failed to get container status \"63afa5a3a93d6f74b869d00e48fb75c41899dd8f639a9117fe3cff87b35472ce\": rpc error: code = NotFound desc = could not find container \"63afa5a3a93d6f74b869d00e48fb75c41899dd8f639a9117fe3cff87b35472ce\": container with ID starting with 63afa5a3a93d6f74b869d00e48fb75c41899dd8f639a9117fe3cff87b35472ce not found: ID does not exist" Oct 29 23:54:31.710792 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.710754 5057 scope.go:122] "RemoveContainer" containerID="c4ce3d9796ce9730a641d772c9bd023d34ec70de5091cb8b3945920278671b4f" Oct 29 23:54:31.720948 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.720928 5057 scope.go:122] "RemoveContainer" containerID="c4ce3d9796ce9730a641d772c9bd023d34ec70de5091cb8b3945920278671b4f" Oct 29 23:54:31.721261 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:54:31.721240 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4ce3d9796ce9730a641d772c9bd023d34ec70de5091cb8b3945920278671b4f\": container with ID starting with c4ce3d9796ce9730a641d772c9bd023d34ec70de5091cb8b3945920278671b4f not found: ID does not exist" containerID="c4ce3d9796ce9730a641d772c9bd023d34ec70de5091cb8b3945920278671b4f" Oct 29 23:54:31.721317 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.721270 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4ce3d9796ce9730a641d772c9bd023d34ec70de5091cb8b3945920278671b4f"} err="failed to get container status \"c4ce3d9796ce9730a641d772c9bd023d34ec70de5091cb8b3945920278671b4f\": rpc error: code = NotFound desc = could not find container \"c4ce3d9796ce9730a641d772c9bd023d34ec70de5091cb8b3945920278671b4f\": container with ID starting with c4ce3d9796ce9730a641d772c9bd023d34ec70de5091cb8b3945920278671b4f not found: ID does not exist" Oct 29 23:54:31.734911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.734879 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29396ecc-dbb0-4fca-a886-99496af5bbd2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7\" (UID: \"29396ecc-dbb0-4fca-a886-99496af5bbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7" Oct 29 23:54:31.734911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.734916 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b9kv\" (UniqueName: \"kubernetes.io/projected/29396ecc-dbb0-4fca-a886-99496af5bbd2-kube-api-access-6b9kv\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7\" (UID: \"29396ecc-dbb0-4fca-a886-99496af5bbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7" Oct 29 23:54:31.735122 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.734976 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29396ecc-dbb0-4fca-a886-99496af5bbd2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7\" (UID: \"29396ecc-dbb0-4fca-a886-99496af5bbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7" Oct 29 23:54:31.735604 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.735584 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29396ecc-dbb0-4fca-a886-99496af5bbd2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7\" (UID: \"29396ecc-dbb0-4fca-a886-99496af5bbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7" Oct 29 23:54:31.737353 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.737331 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29396ecc-dbb0-4fca-a886-99496af5bbd2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7\" (UID: \"29396ecc-dbb0-4fca-a886-99496af5bbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7" Oct 29 23:54:31.750114 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.750086 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr"] Oct 29 23:54:31.754175 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.754147 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2077b4-667df8545b-l2kpr"] Oct 29 23:54:31.758689 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.758663 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b9kv\" (UniqueName: \"kubernetes.io/projected/29396ecc-dbb0-4fca-a886-99496af5bbd2-kube-api-access-6b9kv\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7\" (UID: \"29396ecc-dbb0-4fca-a886-99496af5bbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7" Oct 29 23:54:31.789736 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.789695 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2"] Oct 29 23:54:31.795616 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.795585 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2077b4-6fb5fc9d8c-f2jm2"] Oct 29 23:54:31.906998 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:31.906964 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7" Oct 29 23:54:32.211594 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:32.211539 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7"] Oct 29 23:54:32.213809 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:54:32.213776 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29396ecc_dbb0_4fca_a886_99496af5bbd2.slice/crio-a4316690f74485be65a1dd9cce18ab4677d6ac8e737efc6f0cd67a490d8ea6a9 WatchSource:0}: Error finding container a4316690f74485be65a1dd9cce18ab4677d6ac8e737efc6f0cd67a490d8ea6a9: Status 404 returned error can't find the container with id a4316690f74485be65a1dd9cce18ab4677d6ac8e737efc6f0cd67a490d8ea6a9 Oct 29 23:54:32.710511 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:32.710474 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7" event={"ID":"29396ecc-dbb0-4fca-a886-99496af5bbd2","Type":"ContainerStarted","Data":"e2d791dd353219720f1c6a91c7c279a28899be2b1fce93b0fd02fdf7e9aedee3"} Oct 29 23:54:32.710924 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:32.710525 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7" event={"ID":"29396ecc-dbb0-4fca-a886-99496af5bbd2","Type":"ContainerStarted","Data":"a4316690f74485be65a1dd9cce18ab4677d6ac8e737efc6f0cd67a490d8ea6a9"} Oct 29 23:54:32.751038 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:32.750985 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7" podStartSLOduration=1.7509706619999998 podStartE2EDuration="1.750970662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:54:32.749616655 +0000 UTC m=+4319.401115187" watchObservedRunningTime="2026-10-29 23:54:32.750970662 +0000 UTC m=+4319.402469192" Oct 29 23:54:33.481100 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:33.481068 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87936d4c-796b-4176-b54a-4b520e0b7ce5" path="/var/lib/kubelet/pods/87936d4c-796b-4176-b54a-4b520e0b7ce5/volumes" Oct 29 23:54:33.481543 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:33.481527 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3a0a5b1-c514-4685-a97d-c75ea6882e82" path="/var/lib/kubelet/pods/e3a0a5b1-c514-4685-a97d-c75ea6882e82/volumes" Oct 29 23:54:33.723668 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:33.723624 5057 generic.go:356] "Generic (PLEG): container finished" podID="29396ecc-dbb0-4fca-a886-99496af5bbd2" containerID="e2d791dd353219720f1c6a91c7c279a28899be2b1fce93b0fd02fdf7e9aedee3" exitCode=0 Oct 29 23:54:33.724345 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:33.723826 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7" event={"ID":"29396ecc-dbb0-4fca-a886-99496af5bbd2","Type":"ContainerDied","Data":"e2d791dd353219720f1c6a91c7c279a28899be2b1fce93b0fd02fdf7e9aedee3"} Oct 29 23:54:33.885023 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:33.884986 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8" podUID="6b25834c-7d18-46f7-85ea-c536e5eb3ea0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:54:33.914147 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:54:33.914104 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:54:34.887126 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:34.887100 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7" Oct 29 23:54:34.892072 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:34.892044 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4" podUID="9655e209-6b99-40ea-9602-51eceac7efa9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:54:34.971754 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:34.971712 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29396ecc-dbb0-4fca-a886-99496af5bbd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29396ecc-dbb0-4fca-a886-99496af5bbd2-maas-api-xks-ca\") pod \"29396ecc-dbb0-4fca-a886-99496af5bbd2\" (UID: \"29396ecc-dbb0-4fca-a886-99496af5bbd2\") " Oct 29 23:54:34.971935 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:34.971822 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29396ecc-dbb0-4fca-a886-99496af5bbd2-kube-api-access-6b9kv\" (UniqueName: \"kubernetes.io/projected/29396ecc-dbb0-4fca-a886-99496af5bbd2-kube-api-access-6b9kv\") pod \"29396ecc-dbb0-4fca-a886-99496af5bbd2\" (UID: \"29396ecc-dbb0-4fca-a886-99496af5bbd2\") " Oct 29 23:54:34.971935 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:34.971899 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29396ecc-dbb0-4fca-a886-99496af5bbd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29396ecc-dbb0-4fca-a886-99496af5bbd2-maas-api-openshift-ca\") pod \"29396ecc-dbb0-4fca-a886-99496af5bbd2\" (UID: \"29396ecc-dbb0-4fca-a886-99496af5bbd2\") " Oct 29 23:54:34.972558 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:34.972508 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29396ecc-dbb0-4fca-a886-99496af5bbd2-maas-api-openshift-ca" pod "29396ecc-dbb0-4fca-a886-99496af5bbd2" (UID: "29396ecc-dbb0-4fca-a886-99496af5bbd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:34.974046 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:34.974019 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29396ecc-dbb0-4fca-a886-99496af5bbd2-maas-api-xks-ca" pod "29396ecc-dbb0-4fca-a886-99496af5bbd2" (UID: "29396ecc-dbb0-4fca-a886-99496af5bbd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:54:34.974046 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:34.974031 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29396ecc-dbb0-4fca-a886-99496af5bbd2-kube-api-access-6b9kv" pod "29396ecc-dbb0-4fca-a886-99496af5bbd2" (UID: "29396ecc-dbb0-4fca-a886-99496af5bbd2"). InnerVolumeSpecName "kube-api-access-6b9kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:35.073147 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:35.073067 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29396ecc-dbb0-4fca-a886-99496af5bbd2-maas-api-openshift-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:35.073147 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:35.073096 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29396ecc-dbb0-4fca-a886-99496af5bbd2-maas-api-xks-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:35.073147 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:35.073108 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b9kv\" (UniqueName: \"kubernetes.io/projected/29396ecc-dbb0-4fca-a886-99496af5bbd2-kube-api-access-6b9kv\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:35.740334 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:35.740249 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7" Oct 29 23:54:35.740489 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:35.740250 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7" event={"ID":"29396ecc-dbb0-4fca-a886-99496af5bbd2","Type":"ContainerDied","Data":"a4316690f74485be65a1dd9cce18ab4677d6ac8e737efc6f0cd67a490d8ea6a9"} Oct 29 23:54:35.740489 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:35.740384 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4316690f74485be65a1dd9cce18ab4677d6ac8e737efc6f0cd67a490d8ea6a9" Oct 29 23:54:35.972114 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:35.972074 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:54:36.799455 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:36.799423 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5"] Oct 29 23:54:36.799702 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:36.799678 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" podUID="f7feb618-94bf-4988-abca-4dc4fbd699c7" containerName="istio-proxy" containerID="cri-o://c08ef566f221f4b3fc622be50beba790a0e491d17dcd3cd69a864b16859ba433" gracePeriod=30 Oct 29 23:54:39.180978 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:39.180940 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:54:39.181455 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:39.180989 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" podUID="f7feb618-94bf-4988-abca-4dc4fbd699c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:54:42.077429 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.077351 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 23:54:42.149458 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.149418 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-workload-socket\") pod \"f7feb618-94bf-4988-abca-4dc4fbd699c7\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " Oct 29 23:54:42.149635 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.149520 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7feb618-94bf-4988-abca-4dc4fbd699c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7feb618-94bf-4988-abca-4dc4fbd699c7-istiod-ca-cert\") pod \"f7feb618-94bf-4988-abca-4dc4fbd699c7\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " Oct 29 23:54:42.149635 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.149549 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-workload-certs\") pod \"f7feb618-94bf-4988-abca-4dc4fbd699c7\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " Oct 29 23:54:42.149635 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.149595 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7feb618-94bf-4988-abca-4dc4fbd699c7-kube-api-access-sdc9l\" (UniqueName: \"kubernetes.io/projected/f7feb618-94bf-4988-abca-4dc4fbd699c7-kube-api-access-sdc9l\") pod \"f7feb618-94bf-4988-abca-4dc4fbd699c7\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " Oct 29 23:54:42.149794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.149653 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-podinfo\") pod \"f7feb618-94bf-4988-abca-4dc4fbd699c7\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " Oct 29 23:54:42.149794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.149712 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-credential-socket\") pod \"f7feb618-94bf-4988-abca-4dc4fbd699c7\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " Oct 29 23:54:42.149794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.149737 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-data\") pod \"f7feb618-94bf-4988-abca-4dc4fbd699c7\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " Oct 29 23:54:42.149794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.149735 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-workload-socket" pod "f7feb618-94bf-4988-abca-4dc4fbd699c7" (UID: "f7feb618-94bf-4988-abca-4dc4fbd699c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:42.149794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.149784 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-envoy\") pod \"f7feb618-94bf-4988-abca-4dc4fbd699c7\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " Oct 29 23:54:42.150483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.149857 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-token\" (UniqueName: \"kubernetes.io/projected/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-token\") pod \"f7feb618-94bf-4988-abca-4dc4fbd699c7\" (UID: \"f7feb618-94bf-4988-abca-4dc4fbd699c7\") " Oct 29 23:54:42.150483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.149904 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-workload-certs" pod "f7feb618-94bf-4988-abca-4dc4fbd699c7" (UID: "f7feb618-94bf-4988-abca-4dc4fbd699c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:42.150483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.150009 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-credential-socket" pod "f7feb618-94bf-4988-abca-4dc4fbd699c7" (UID: "f7feb618-94bf-4988-abca-4dc4fbd699c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:42.150483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.149937 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7feb618-94bf-4988-abca-4dc4fbd699c7-istiod-ca-cert" pod "f7feb618-94bf-4988-abca-4dc4fbd699c7" (UID: "f7feb618-94bf-4988-abca-4dc4fbd699c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:42.150483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.150100 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-data" pod "f7feb618-94bf-4988-abca-4dc4fbd699c7" (UID: "f7feb618-94bf-4988-abca-4dc4fbd699c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:42.150483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.150287 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:42.150483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.150306 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:42.150483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.150320 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:42.150483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.150332 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7feb618-94bf-4988-abca-4dc4fbd699c7-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:42.150483 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.150345 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:42.151999 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.151976 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-envoy" pod "f7feb618-94bf-4988-abca-4dc4fbd699c7" (UID: "f7feb618-94bf-4988-abca-4dc4fbd699c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:42.152254 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.152225 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7feb618-94bf-4988-abca-4dc4fbd699c7-kube-api-access-sdc9l" pod "f7feb618-94bf-4988-abca-4dc4fbd699c7" (UID: "f7feb618-94bf-4988-abca-4dc4fbd699c7"). InnerVolumeSpecName "kube-api-access-sdc9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:42.152450 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.152425 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-token" pod "f7feb618-94bf-4988-abca-4dc4fbd699c7" (UID: "f7feb618-94bf-4988-abca-4dc4fbd699c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:42.152907 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.152880 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-podinfo" pod "f7feb618-94bf-4988-abca-4dc4fbd699c7" (UID: "f7feb618-94bf-4988-abca-4dc4fbd699c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:54:42.251930 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.251896 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdc9l\" (UniqueName: \"kubernetes.io/projected/f7feb618-94bf-4988-abca-4dc4fbd699c7-kube-api-access-sdc9l\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:42.251930 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.251929 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:42.251930 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.251940 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:42.252180 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.251948 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7feb618-94bf-4988-abca-4dc4fbd699c7-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:42.782798 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.782761 5057 generic.go:356] "Generic (PLEG): container finished" podID="f7feb618-94bf-4988-abca-4dc4fbd699c7" containerID="c08ef566f221f4b3fc622be50beba790a0e491d17dcd3cd69a864b16859ba433" exitCode=0 Oct 29 23:54:42.782976 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.782820 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" Oct 29 23:54:42.782976 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.782854 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" event={"ID":"f7feb618-94bf-4988-abca-4dc4fbd699c7","Type":"ContainerDied","Data":"c08ef566f221f4b3fc622be50beba790a0e491d17dcd3cd69a864b16859ba433"} Oct 29 23:54:42.782976 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.782926 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5" event={"ID":"f7feb618-94bf-4988-abca-4dc4fbd699c7","Type":"ContainerDied","Data":"38565599b99ba66e7a7fc79593e8da418ee184452d9fbd791f0ca292fd1983f5"} Oct 29 23:54:42.782976 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.782960 5057 scope.go:122] "RemoveContainer" containerID="c08ef566f221f4b3fc622be50beba790a0e491d17dcd3cd69a864b16859ba433" Oct 29 23:54:42.793449 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.793432 5057 scope.go:122] "RemoveContainer" containerID="c24e4bee009e74b6b1fc9cffd573f1b02858720006716f2c2ce40d763626c965" Oct 29 23:54:42.801984 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.801961 5057 scope.go:122] "RemoveContainer" containerID="c08ef566f221f4b3fc622be50beba790a0e491d17dcd3cd69a864b16859ba433" Oct 29 23:54:42.802273 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:54:42.802256 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c08ef566f221f4b3fc622be50beba790a0e491d17dcd3cd69a864b16859ba433\": container with ID starting with c08ef566f221f4b3fc622be50beba790a0e491d17dcd3cd69a864b16859ba433 not found: ID does not exist" containerID="c08ef566f221f4b3fc622be50beba790a0e491d17dcd3cd69a864b16859ba433" Oct 29 23:54:42.802348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.802281 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c08ef566f221f4b3fc622be50beba790a0e491d17dcd3cd69a864b16859ba433"} err="failed to get container status \"c08ef566f221f4b3fc622be50beba790a0e491d17dcd3cd69a864b16859ba433\": rpc error: code = NotFound desc = could not find container \"c08ef566f221f4b3fc622be50beba790a0e491d17dcd3cd69a864b16859ba433\": container with ID starting with c08ef566f221f4b3fc622be50beba790a0e491d17dcd3cd69a864b16859ba433 not found: ID does not exist" Oct 29 23:54:42.802348 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.802301 5057 scope.go:122] "RemoveContainer" containerID="c24e4bee009e74b6b1fc9cffd573f1b02858720006716f2c2ce40d763626c965" Oct 29 23:54:42.802529 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:54:42.802516 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c24e4bee009e74b6b1fc9cffd573f1b02858720006716f2c2ce40d763626c965\": container with ID starting with c24e4bee009e74b6b1fc9cffd573f1b02858720006716f2c2ce40d763626c965 not found: ID does not exist" containerID="c24e4bee009e74b6b1fc9cffd573f1b02858720006716f2c2ce40d763626c965" Oct 29 23:54:42.802569 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.802533 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24e4bee009e74b6b1fc9cffd573f1b02858720006716f2c2ce40d763626c965"} err="failed to get container status \"c24e4bee009e74b6b1fc9cffd573f1b02858720006716f2c2ce40d763626c965\": rpc error: code = NotFound desc = could not find container \"c24e4bee009e74b6b1fc9cffd573f1b02858720006716f2c2ce40d763626c965\": container with ID starting with c24e4bee009e74b6b1fc9cffd573f1b02858720006716f2c2ce40d763626c965 not found: ID does not exist" Oct 29 23:54:42.846836 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.846801 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5"] Oct 29 23:54:42.860139 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:42.860110 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-ksrz5"] Oct 29 23:54:43.480597 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:43.480553 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7feb618-94bf-4988-abca-4dc4fbd699c7" path="/var/lib/kubelet/pods/f7feb618-94bf-4988-abca-4dc4fbd699c7/volumes" Oct 29 23:54:43.884352 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:43.884260 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8" podUID="6b25834c-7d18-46f7-85ea-c536e5eb3ea0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:54:44.705768 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:44.705718 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7"] Oct 29 23:54:44.716734 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:44.716699 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8a8b61-2zcd7"] Oct 29 23:54:44.892500 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:44.892467 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4" podUID="9655e209-6b99-40ea-9602-51eceac7efa9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:54:45.480599 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:45.480562 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29396ecc-dbb0-4fca-a886-99496af5bbd2" path="/var/lib/kubelet/pods/29396ecc-dbb0-4fca-a886-99496af5bbd2/volumes" Oct 29 23:54:48.599657 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.599627 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8" Oct 29 23:54:48.625739 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.625715 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4" Oct 29 23:54:48.727074 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.727043 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9655e209-6b99-40ea-9602-51eceac7efa9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9655e209-6b99-40ea-9602-51eceac7efa9-plugins-config-volume\") pod \"9655e209-6b99-40ea-9602-51eceac7efa9\" (UID: \"9655e209-6b99-40ea-9602-51eceac7efa9\") " Oct 29 23:54:48.727264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.727082 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b25834c-7d18-46f7-85ea-c536e5eb3ea0-kube-api-access-8lsdk\" (UniqueName: \"kubernetes.io/projected/6b25834c-7d18-46f7-85ea-c536e5eb3ea0-kube-api-access-8lsdk\") pod \"6b25834c-7d18-46f7-85ea-c536e5eb3ea0\" (UID: \"6b25834c-7d18-46f7-85ea-c536e5eb3ea0\") " Oct 29 23:54:48.727264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.727160 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9655e209-6b99-40ea-9602-51eceac7efa9-kube-api-access-bpr6t\" (UniqueName: \"kubernetes.io/projected/9655e209-6b99-40ea-9602-51eceac7efa9-kube-api-access-bpr6t\") pod \"9655e209-6b99-40ea-9602-51eceac7efa9\" (UID: \"9655e209-6b99-40ea-9602-51eceac7efa9\") " Oct 29 23:54:48.727264 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.727232 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b25834c-7d18-46f7-85ea-c536e5eb3ea0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b25834c-7d18-46f7-85ea-c536e5eb3ea0-plugins-config-volume\") pod \"6b25834c-7d18-46f7-85ea-c536e5eb3ea0\" (UID: \"6b25834c-7d18-46f7-85ea-c536e5eb3ea0\") " Oct 29 23:54:48.727653 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.727505 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9655e209-6b99-40ea-9602-51eceac7efa9-plugins-config-volume" pod "9655e209-6b99-40ea-9602-51eceac7efa9" (UID: "9655e209-6b99-40ea-9602-51eceac7efa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:48.727741 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.727669 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b25834c-7d18-46f7-85ea-c536e5eb3ea0-plugins-config-volume" pod "6b25834c-7d18-46f7-85ea-c536e5eb3ea0" (UID: "6b25834c-7d18-46f7-85ea-c536e5eb3ea0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:48.729404 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.729375 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b25834c-7d18-46f7-85ea-c536e5eb3ea0-kube-api-access-8lsdk" pod "6b25834c-7d18-46f7-85ea-c536e5eb3ea0" (UID: "6b25834c-7d18-46f7-85ea-c536e5eb3ea0"). InnerVolumeSpecName "kube-api-access-8lsdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:48.729502 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.729410 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9655e209-6b99-40ea-9602-51eceac7efa9-kube-api-access-bpr6t" pod "9655e209-6b99-40ea-9602-51eceac7efa9" (UID: "9655e209-6b99-40ea-9602-51eceac7efa9"). InnerVolumeSpecName "kube-api-access-bpr6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:48.817736 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.817660 5057 generic.go:356] "Generic (PLEG): container finished" podID="9655e209-6b99-40ea-9602-51eceac7efa9" containerID="8b2228c1e639b8054be991efcca4d2d95c608ffc068e1ccfc47aa8664cf6b69d" exitCode=0 Oct 29 23:54:48.817736 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.817724 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4" Oct 29 23:54:48.817922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.817757 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4" event={"ID":"9655e209-6b99-40ea-9602-51eceac7efa9","Type":"ContainerDied","Data":"8b2228c1e639b8054be991efcca4d2d95c608ffc068e1ccfc47aa8664cf6b69d"} Oct 29 23:54:48.817922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.817814 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4" event={"ID":"9655e209-6b99-40ea-9602-51eceac7efa9","Type":"ContainerDied","Data":"e9b1508603c20a6333af87849331750439490240841469442b41550e47882c71"} Oct 29 23:54:48.817922 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.817840 5057 scope.go:122] "RemoveContainer" containerID="8b2228c1e639b8054be991efcca4d2d95c608ffc068e1ccfc47aa8664cf6b69d" Oct 29 23:54:48.819764 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.819733 5057 generic.go:356] "Generic (PLEG): container finished" podID="6b25834c-7d18-46f7-85ea-c536e5eb3ea0" containerID="3ea3cbe3dfeab5910d6a99e2ecb14c932783eae7212599dcd0df27f1658368db" exitCode=0 Oct 29 23:54:48.819900 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.819766 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8" event={"ID":"6b25834c-7d18-46f7-85ea-c536e5eb3ea0","Type":"ContainerDied","Data":"3ea3cbe3dfeab5910d6a99e2ecb14c932783eae7212599dcd0df27f1658368db"} Oct 29 23:54:48.819900 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.819796 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8" Oct 29 23:54:48.819900 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.819818 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8" event={"ID":"6b25834c-7d18-46f7-85ea-c536e5eb3ea0","Type":"ContainerDied","Data":"d9444a1c6577dd0e04dbad858c90b56f43c52beb5f3da4f529fe294fa4536037"} Oct 29 23:54:48.828066 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.828010 5057 scope.go:122] "RemoveContainer" containerID="8b2228c1e639b8054be991efcca4d2d95c608ffc068e1ccfc47aa8664cf6b69d" Oct 29 23:54:48.828353 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:54:48.828321 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2228c1e639b8054be991efcca4d2d95c608ffc068e1ccfc47aa8664cf6b69d\": container with ID starting with 8b2228c1e639b8054be991efcca4d2d95c608ffc068e1ccfc47aa8664cf6b69d not found: ID does not exist" containerID="8b2228c1e639b8054be991efcca4d2d95c608ffc068e1ccfc47aa8664cf6b69d" Oct 29 23:54:48.828446 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.828365 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2228c1e639b8054be991efcca4d2d95c608ffc068e1ccfc47aa8664cf6b69d"} err="failed to get container status \"8b2228c1e639b8054be991efcca4d2d95c608ffc068e1ccfc47aa8664cf6b69d\": rpc error: code = NotFound desc = could not find container \"8b2228c1e639b8054be991efcca4d2d95c608ffc068e1ccfc47aa8664cf6b69d\": container with ID starting with 8b2228c1e639b8054be991efcca4d2d95c608ffc068e1ccfc47aa8664cf6b69d not found: ID does not exist" Oct 29 23:54:48.828446 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.828390 5057 scope.go:122] "RemoveContainer" containerID="3ea3cbe3dfeab5910d6a99e2ecb14c932783eae7212599dcd0df27f1658368db" Oct 29 23:54:48.828566 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.828534 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9655e209-6b99-40ea-9602-51eceac7efa9-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:48.828566 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.828554 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lsdk\" (UniqueName: \"kubernetes.io/projected/6b25834c-7d18-46f7-85ea-c536e5eb3ea0-kube-api-access-8lsdk\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:48.828566 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.828562 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpr6t\" (UniqueName: \"kubernetes.io/projected/9655e209-6b99-40ea-9602-51eceac7efa9-kube-api-access-bpr6t\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:48.828890 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.828571 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b25834c-7d18-46f7-85ea-c536e5eb3ea0-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:48.838307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.838285 5057 scope.go:122] "RemoveContainer" containerID="3ea3cbe3dfeab5910d6a99e2ecb14c932783eae7212599dcd0df27f1658368db" Oct 29 23:54:48.838588 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:54:48.838569 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea3cbe3dfeab5910d6a99e2ecb14c932783eae7212599dcd0df27f1658368db\": container with ID starting with 3ea3cbe3dfeab5910d6a99e2ecb14c932783eae7212599dcd0df27f1658368db not found: ID does not exist" containerID="3ea3cbe3dfeab5910d6a99e2ecb14c932783eae7212599dcd0df27f1658368db" Oct 29 23:54:48.838659 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.838604 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea3cbe3dfeab5910d6a99e2ecb14c932783eae7212599dcd0df27f1658368db"} err="failed to get container status \"3ea3cbe3dfeab5910d6a99e2ecb14c932783eae7212599dcd0df27f1658368db\": rpc error: code = NotFound desc = could not find container \"3ea3cbe3dfeab5910d6a99e2ecb14c932783eae7212599dcd0df27f1658368db\": container with ID starting with 3ea3cbe3dfeab5910d6a99e2ecb14c932783eae7212599dcd0df27f1658368db not found: ID does not exist" Oct 29 23:54:48.869161 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.869125 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4"] Oct 29 23:54:48.873578 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.873557 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-32c192-8655bfc999-tzxn4"] Oct 29 23:54:48.907125 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.907102 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8"] Oct 29 23:54:48.910773 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:48.910743 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-32c192-6bfbc897f6-wp2f8"] Oct 29 23:54:49.480596 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:49.480563 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b25834c-7d18-46f7-85ea-c536e5eb3ea0" path="/var/lib/kubelet/pods/6b25834c-7d18-46f7-85ea-c536e5eb3ea0/volumes" Oct 29 23:54:49.480956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:49.480943 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9655e209-6b99-40ea-9602-51eceac7efa9" path="/var/lib/kubelet/pods/9655e209-6b99-40ea-9602-51eceac7efa9/volumes" Oct 29 23:54:51.713216 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:51.713163 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t"] Oct 29 23:54:51.713724 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:51.713436 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" podUID="db576741-edce-4ad9-8f81-0d8a8f110934" containerName="istio-proxy" containerID="cri-o://89f43b34408003a349700c605cc515a4cf275d747478e6c10ce45f3c8d65b6d0" gracePeriod=30 Oct 29 23:54:52.477715 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:52.477679 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd"] Oct 29 23:54:52.477953 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:52.477928 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" podUID="408b4a17-ac05-47f5-bbcb-ed398229570c" containerName="istio-proxy" containerID="cri-o://276620c57c73fdf3f28ccb85e9fabbf67a5d7b69bfee33ca4d81a2f01fe4a955" gracePeriod=30 Oct 29 23:54:54.891702 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:54.891668 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:54:55.422583 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:55.422546 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:54:55.422766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:55.422626 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" podUID="db576741-edce-4ad9-8f81-0d8a8f110934" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:54:56.871256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:56.871219 5057 generic.go:356] "Generic (PLEG): container finished" podID="db576741-edce-4ad9-8f81-0d8a8f110934" containerID="89f43b34408003a349700c605cc515a4cf275d747478e6c10ce45f3c8d65b6d0" exitCode=0 Oct 29 23:54:56.871642 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:56.871232 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" event={"ID":"db576741-edce-4ad9-8f81-0d8a8f110934","Type":"ContainerDied","Data":"89f43b34408003a349700c605cc515a4cf275d747478e6c10ce45f3c8d65b6d0"} Oct 29 23:54:56.990544 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:56.990491 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 23:54:57.114088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.114053 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-istio-envoy\") pod \"db576741-edce-4ad9-8f81-0d8a8f110934\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " Oct 29 23:54:57.114282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.114100 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-istio-data\") pod \"db576741-edce-4ad9-8f81-0d8a8f110934\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " Oct 29 23:54:57.114282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.114217 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-workload-certs\") pod \"db576741-edce-4ad9-8f81-0d8a8f110934\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " Oct 29 23:54:57.114282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.114262 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-credential-socket\") pod \"db576741-edce-4ad9-8f81-0d8a8f110934\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " Oct 29 23:54:57.114471 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.114321 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db576741-edce-4ad9-8f81-0d8a8f110934-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db576741-edce-4ad9-8f81-0d8a8f110934-istiod-ca-cert\") pod \"db576741-edce-4ad9-8f81-0d8a8f110934\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " Oct 29 23:54:57.114471 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.114357 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db576741-edce-4ad9-8f81-0d8a8f110934-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db576741-edce-4ad9-8f81-0d8a8f110934-istio-podinfo\") pod \"db576741-edce-4ad9-8f81-0d8a8f110934\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " Oct 29 23:54:57.114471 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.114395 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db576741-edce-4ad9-8f81-0d8a8f110934-kube-api-access-n4d4w\" (UniqueName: \"kubernetes.io/projected/db576741-edce-4ad9-8f81-0d8a8f110934-kube-api-access-n4d4w\") pod \"db576741-edce-4ad9-8f81-0d8a8f110934\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " Oct 29 23:54:57.114471 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.114424 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-workload-socket\") pod \"db576741-edce-4ad9-8f81-0d8a8f110934\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " Oct 29 23:54:57.114471 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.114437 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-istio-data" pod "db576741-edce-4ad9-8f81-0d8a8f110934" (UID: "db576741-edce-4ad9-8f81-0d8a8f110934"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:57.114771 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.114483 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db576741-edce-4ad9-8f81-0d8a8f110934-istio-token\" (UniqueName: \"kubernetes.io/projected/db576741-edce-4ad9-8f81-0d8a8f110934-istio-token\") pod \"db576741-edce-4ad9-8f81-0d8a8f110934\" (UID: \"db576741-edce-4ad9-8f81-0d8a8f110934\") " Oct 29 23:54:57.114771 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.114547 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-workload-certs" pod "db576741-edce-4ad9-8f81-0d8a8f110934" (UID: "db576741-edce-4ad9-8f81-0d8a8f110934"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:57.114771 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.114629 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-credential-socket" pod "db576741-edce-4ad9-8f81-0d8a8f110934" (UID: "db576741-edce-4ad9-8f81-0d8a8f110934"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:57.114771 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.114742 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db576741-edce-4ad9-8f81-0d8a8f110934-istiod-ca-cert" pod "db576741-edce-4ad9-8f81-0d8a8f110934" (UID: "db576741-edce-4ad9-8f81-0d8a8f110934"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:57.115001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.114911 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.115001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.114929 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.115001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.114945 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.115001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.114958 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db576741-edce-4ad9-8f81-0d8a8f110934-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.115001 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.114932 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-workload-socket" pod "db576741-edce-4ad9-8f81-0d8a8f110934" (UID: "db576741-edce-4ad9-8f81-0d8a8f110934"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:57.116937 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.116911 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db576741-edce-4ad9-8f81-0d8a8f110934-istio-token" pod "db576741-edce-4ad9-8f81-0d8a8f110934" (UID: "db576741-edce-4ad9-8f81-0d8a8f110934"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:57.117064 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.116942 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db576741-edce-4ad9-8f81-0d8a8f110934-kube-api-access-n4d4w" pod "db576741-edce-4ad9-8f81-0d8a8f110934" (UID: "db576741-edce-4ad9-8f81-0d8a8f110934"). InnerVolumeSpecName "kube-api-access-n4d4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:57.117155 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.117135 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db576741-edce-4ad9-8f81-0d8a8f110934-istio-podinfo" pod "db576741-edce-4ad9-8f81-0d8a8f110934" (UID: "db576741-edce-4ad9-8f81-0d8a8f110934"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:54:57.117608 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.117581 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-istio-envoy" pod "db576741-edce-4ad9-8f81-0d8a8f110934" (UID: "db576741-edce-4ad9-8f81-0d8a8f110934"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:57.216398 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.216366 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db576741-edce-4ad9-8f81-0d8a8f110934-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.216398 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.216393 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4d4w\" (UniqueName: \"kubernetes.io/projected/db576741-edce-4ad9-8f81-0d8a8f110934-kube-api-access-n4d4w\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.216398 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.216402 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.216636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.216412 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db576741-edce-4ad9-8f81-0d8a8f110934-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.216636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.216423 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db576741-edce-4ad9-8f81-0d8a8f110934-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.726461 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.726436 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:54:57.822048 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.822017 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-data\") pod \"408b4a17-ac05-47f5-bbcb-ed398229570c\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " Oct 29 23:54:57.822048 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.822053 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-token\" (UniqueName: \"kubernetes.io/projected/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-token\") pod \"408b4a17-ac05-47f5-bbcb-ed398229570c\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " Oct 29 23:54:57.822520 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.822117 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-podinfo\") pod \"408b4a17-ac05-47f5-bbcb-ed398229570c\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " Oct 29 23:54:57.822520 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.822182 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-envoy\") pod \"408b4a17-ac05-47f5-bbcb-ed398229570c\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " Oct 29 23:54:57.822520 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.822247 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408b4a17-ac05-47f5-bbcb-ed398229570c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408b4a17-ac05-47f5-bbcb-ed398229570c-istiod-ca-cert\") pod \"408b4a17-ac05-47f5-bbcb-ed398229570c\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " Oct 29 23:54:57.822520 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.822312 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-workload-certs\") pod \"408b4a17-ac05-47f5-bbcb-ed398229570c\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " Oct 29 23:54:57.822520 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.822316 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-data" pod "408b4a17-ac05-47f5-bbcb-ed398229570c" (UID: "408b4a17-ac05-47f5-bbcb-ed398229570c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:57.822520 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.822339 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-workload-socket\") pod \"408b4a17-ac05-47f5-bbcb-ed398229570c\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " Oct 29 23:54:57.822520 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.822368 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408b4a17-ac05-47f5-bbcb-ed398229570c-kube-api-access-wvr8c\" (UniqueName: \"kubernetes.io/projected/408b4a17-ac05-47f5-bbcb-ed398229570c-kube-api-access-wvr8c\") pod \"408b4a17-ac05-47f5-bbcb-ed398229570c\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " Oct 29 23:54:57.822520 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.822410 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-credential-socket\") pod \"408b4a17-ac05-47f5-bbcb-ed398229570c\" (UID: \"408b4a17-ac05-47f5-bbcb-ed398229570c\") " Oct 29 23:54:57.823099 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.822801 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408b4a17-ac05-47f5-bbcb-ed398229570c-istiod-ca-cert" pod "408b4a17-ac05-47f5-bbcb-ed398229570c" (UID: "408b4a17-ac05-47f5-bbcb-ed398229570c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:57.823099 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.822894 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.823099 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.822942 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-workload-socket" pod "408b4a17-ac05-47f5-bbcb-ed398229570c" (UID: "408b4a17-ac05-47f5-bbcb-ed398229570c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:57.823099 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.823031 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-workload-certs" pod "408b4a17-ac05-47f5-bbcb-ed398229570c" (UID: "408b4a17-ac05-47f5-bbcb-ed398229570c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:57.823355 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.823114 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-credential-socket" pod "408b4a17-ac05-47f5-bbcb-ed398229570c" (UID: "408b4a17-ac05-47f5-bbcb-ed398229570c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:57.824770 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.824741 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-envoy" pod "408b4a17-ac05-47f5-bbcb-ed398229570c" (UID: "408b4a17-ac05-47f5-bbcb-ed398229570c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:57.824870 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.824849 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-podinfo" pod "408b4a17-ac05-47f5-bbcb-ed398229570c" (UID: "408b4a17-ac05-47f5-bbcb-ed398229570c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:54:57.824966 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.824949 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408b4a17-ac05-47f5-bbcb-ed398229570c-kube-api-access-wvr8c" pod "408b4a17-ac05-47f5-bbcb-ed398229570c" (UID: "408b4a17-ac05-47f5-bbcb-ed398229570c"). InnerVolumeSpecName "kube-api-access-wvr8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:57.825029 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.825007 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-token" pod "408b4a17-ac05-47f5-bbcb-ed398229570c" (UID: "408b4a17-ac05-47f5-bbcb-ed398229570c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:57.878705 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.878669 5057 generic.go:356] "Generic (PLEG): container finished" podID="408b4a17-ac05-47f5-bbcb-ed398229570c" containerID="276620c57c73fdf3f28ccb85e9fabbf67a5d7b69bfee33ca4d81a2f01fe4a955" exitCode=0 Oct 29 23:54:57.879104 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.878714 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" Oct 29 23:54:57.879104 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.878750 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" event={"ID":"408b4a17-ac05-47f5-bbcb-ed398229570c","Type":"ContainerDied","Data":"276620c57c73fdf3f28ccb85e9fabbf67a5d7b69bfee33ca4d81a2f01fe4a955"} Oct 29 23:54:57.879104 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.878814 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd" event={"ID":"408b4a17-ac05-47f5-bbcb-ed398229570c","Type":"ContainerDied","Data":"33197e17df780cd9fb3713a9d2491cc5816abf25c2df0b7947ea589ad529ef5a"} Oct 29 23:54:57.879104 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.878837 5057 scope.go:122] "RemoveContainer" containerID="276620c57c73fdf3f28ccb85e9fabbf67a5d7b69bfee33ca4d81a2f01fe4a955" Oct 29 23:54:57.880402 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.880380 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" event={"ID":"db576741-edce-4ad9-8f81-0d8a8f110934","Type":"ContainerDied","Data":"22e7cbcc1100b07755cac3fa6bf9f10193f622c07a22d8c115672bba5637aac0"} Oct 29 23:54:57.880402 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.880399 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t" Oct 29 23:54:57.890307 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.890292 5057 scope.go:122] "RemoveContainer" containerID="0abed984960160f246c3f1ddf058d37cfd85832fe262ba396e07da3a699545ce" Oct 29 23:54:57.898719 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.898699 5057 scope.go:122] "RemoveContainer" containerID="276620c57c73fdf3f28ccb85e9fabbf67a5d7b69bfee33ca4d81a2f01fe4a955" Oct 29 23:54:57.899004 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:54:57.898985 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"276620c57c73fdf3f28ccb85e9fabbf67a5d7b69bfee33ca4d81a2f01fe4a955\": container with ID starting with 276620c57c73fdf3f28ccb85e9fabbf67a5d7b69bfee33ca4d81a2f01fe4a955 not found: ID does not exist" containerID="276620c57c73fdf3f28ccb85e9fabbf67a5d7b69bfee33ca4d81a2f01fe4a955" Oct 29 23:54:57.899070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.899013 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"276620c57c73fdf3f28ccb85e9fabbf67a5d7b69bfee33ca4d81a2f01fe4a955"} err="failed to get container status \"276620c57c73fdf3f28ccb85e9fabbf67a5d7b69bfee33ca4d81a2f01fe4a955\": rpc error: code = NotFound desc = could not find container \"276620c57c73fdf3f28ccb85e9fabbf67a5d7b69bfee33ca4d81a2f01fe4a955\": container with ID starting with 276620c57c73fdf3f28ccb85e9fabbf67a5d7b69bfee33ca4d81a2f01fe4a955 not found: ID does not exist" Oct 29 23:54:57.899070 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.899030 5057 scope.go:122] "RemoveContainer" containerID="0abed984960160f246c3f1ddf058d37cfd85832fe262ba396e07da3a699545ce" Oct 29 23:54:57.899332 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:54:57.899312 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0abed984960160f246c3f1ddf058d37cfd85832fe262ba396e07da3a699545ce\": container with ID starting with 0abed984960160f246c3f1ddf058d37cfd85832fe262ba396e07da3a699545ce not found: ID does not exist" containerID="0abed984960160f246c3f1ddf058d37cfd85832fe262ba396e07da3a699545ce" Oct 29 23:54:57.899391 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.899338 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0abed984960160f246c3f1ddf058d37cfd85832fe262ba396e07da3a699545ce"} err="failed to get container status \"0abed984960160f246c3f1ddf058d37cfd85832fe262ba396e07da3a699545ce\": rpc error: code = NotFound desc = could not find container \"0abed984960160f246c3f1ddf058d37cfd85832fe262ba396e07da3a699545ce\": container with ID starting with 0abed984960160f246c3f1ddf058d37cfd85832fe262ba396e07da3a699545ce not found: ID does not exist" Oct 29 23:54:57.899391 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.899352 5057 scope.go:122] "RemoveContainer" containerID="89f43b34408003a349700c605cc515a4cf275d747478e6c10ce45f3c8d65b6d0" Oct 29 23:54:57.923915 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.923893 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.923915 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.923913 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.924064 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.923922 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408b4a17-ac05-47f5-bbcb-ed398229570c-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.924064 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.923930 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.924064 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.923938 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.924064 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.923946 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvr8c\" (UniqueName: \"kubernetes.io/projected/408b4a17-ac05-47f5-bbcb-ed398229570c-kube-api-access-wvr8c\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.924064 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.923956 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408b4a17-ac05-47f5-bbcb-ed398229570c-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.924064 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.923965 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408b4a17-ac05-47f5-bbcb-ed398229570c-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.930325 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.930303 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd"] Oct 29 23:54:57.933911 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.933886 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-bx2gd"] Oct 29 23:54:57.944622 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.944594 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l"] Oct 29 23:54:57.972749 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.972722 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t"] Oct 29 23:54:57.982718 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:57.982691 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-wsj8t"] Oct 29 23:54:59.479857 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:59.479811 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408b4a17-ac05-47f5-bbcb-ed398229570c" path="/var/lib/kubelet/pods/408b4a17-ac05-47f5-bbcb-ed398229570c/volumes" Oct 29 23:54:59.480403 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:54:59.480371 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db576741-edce-4ad9-8f81-0d8a8f110934" path="/var/lib/kubelet/pods/db576741-edce-4ad9-8f81-0d8a8f110934/volumes" Oct 29 23:55:03.640297 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:03.640259 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh"] Oct 29 23:55:03.912651 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:55:03.912564 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:55:04.466800 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.466767 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2"] Oct 29 23:55:04.467914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.467878 5057 cpu_manager.go:414] "containerMap: removing container" podUID="408b4a17-ac05-47f5-bbcb-ed398229570c" containerName="istio-proxy" Oct 29 23:55:04.467914 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.467915 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408b4a17-ac05-47f5-bbcb-ed398229570c" containerName="istio-proxy" Oct 29 23:55:04.468088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.467957 5057 cpu_manager.go:414] "containerMap: removing container" podUID="6b25834c-7d18-46f7-85ea-c536e5eb3ea0" containerName="payload-processing" Oct 29 23:55:04.468088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.467965 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b25834c-7d18-46f7-85ea-c536e5eb3ea0" containerName="payload-processing" Oct 29 23:55:04.468088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.468003 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f7feb618-94bf-4988-abca-4dc4fbd699c7" containerName="istio-proxy" Oct 29 23:55:04.468088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.468011 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7feb618-94bf-4988-abca-4dc4fbd699c7" containerName="istio-proxy" Oct 29 23:55:04.468088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.468038 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9655e209-6b99-40ea-9602-51eceac7efa9" containerName="payload-pre-processing" Oct 29 23:55:04.468088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.468046 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9655e209-6b99-40ea-9602-51eceac7efa9" containerName="payload-pre-processing" Oct 29 23:55:04.468088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.468060 5057 cpu_manager.go:414] "containerMap: removing container" podUID="29396ecc-dbb0-4fca-a886-99496af5bbd2" containerName="revoke-keys" Oct 29 23:55:04.468088 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.468068 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29396ecc-dbb0-4fca-a886-99496af5bbd2" containerName="revoke-keys" Oct 29 23:55:04.468529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.468094 5057 cpu_manager.go:414] "containerMap: removing container" podUID="db576741-edce-4ad9-8f81-0d8a8f110934" containerName="istio-proxy" Oct 29 23:55:04.468529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.468101 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db576741-edce-4ad9-8f81-0d8a8f110934" containerName="istio-proxy" Oct 29 23:55:04.468529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.468337 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="408b4a17-ac05-47f5-bbcb-ed398229570c" containerName="istio-proxy" Oct 29 23:55:04.468529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.468352 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b25834c-7d18-46f7-85ea-c536e5eb3ea0" containerName="payload-processing" Oct 29 23:55:04.468529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.468365 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7feb618-94bf-4988-abca-4dc4fbd699c7" containerName="istio-proxy" Oct 29 23:55:04.468529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.468382 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="408b4a17-ac05-47f5-bbcb-ed398229570c" containerName="istio-proxy" Oct 29 23:55:04.468529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.468397 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9655e209-6b99-40ea-9602-51eceac7efa9" containerName="payload-pre-processing" Oct 29 23:55:04.468529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.468407 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="29396ecc-dbb0-4fca-a886-99496af5bbd2" containerName="revoke-keys" Oct 29 23:55:04.468529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.468422 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7feb618-94bf-4988-abca-4dc4fbd699c7" containerName="istio-proxy" Oct 29 23:55:04.468529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.468430 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="db576741-edce-4ad9-8f81-0d8a8f110934" containerName="istio-proxy" Oct 29 23:55:04.475687 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.475657 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2" Oct 29 23:55:04.493907 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.493878 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2"] Oct 29 23:55:04.574977 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.574942 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:55:04.574977 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.574981 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:55:04.593434 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.593400 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7bb5b36-d921-496a-8c03-953a2645aec5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-271728b7-nwcx2\" (UID: \"b7bb5b36-d921-496a-8c03-953a2645aec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2" Oct 29 23:55:04.593602 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.593457 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-626pg\" (UniqueName: \"kubernetes.io/projected/b7bb5b36-d921-496a-8c03-953a2645aec5-kube-api-access-626pg\") pod \"maas-api-revoke-sub-271728b7-nwcx2\" (UID: \"b7bb5b36-d921-496a-8c03-953a2645aec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2" Oct 29 23:55:04.593681 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.593596 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7bb5b36-d921-496a-8c03-953a2645aec5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-271728b7-nwcx2\" (UID: \"b7bb5b36-d921-496a-8c03-953a2645aec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2" Oct 29 23:55:04.695030 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.694996 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7bb5b36-d921-496a-8c03-953a2645aec5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-271728b7-nwcx2\" (UID: \"b7bb5b36-d921-496a-8c03-953a2645aec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2" Oct 29 23:55:04.695510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.695075 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7bb5b36-d921-496a-8c03-953a2645aec5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-271728b7-nwcx2\" (UID: \"b7bb5b36-d921-496a-8c03-953a2645aec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2" Oct 29 23:55:04.695510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.695158 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-626pg\" (UniqueName: \"kubernetes.io/projected/b7bb5b36-d921-496a-8c03-953a2645aec5-kube-api-access-626pg\") pod \"maas-api-revoke-sub-271728b7-nwcx2\" (UID: \"b7bb5b36-d921-496a-8c03-953a2645aec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2" Oct 29 23:55:04.695939 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.695910 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7bb5b36-d921-496a-8c03-953a2645aec5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-271728b7-nwcx2\" (UID: \"b7bb5b36-d921-496a-8c03-953a2645aec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2" Oct 29 23:55:04.697513 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.697491 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7bb5b36-d921-496a-8c03-953a2645aec5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-271728b7-nwcx2\" (UID: \"b7bb5b36-d921-496a-8c03-953a2645aec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2" Oct 29 23:55:04.718125 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.718071 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-626pg\" (UniqueName: \"kubernetes.io/projected/b7bb5b36-d921-496a-8c03-953a2645aec5-kube-api-access-626pg\") pod \"maas-api-revoke-sub-271728b7-nwcx2\" (UID: \"b7bb5b36-d921-496a-8c03-953a2645aec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2" Oct 29 23:55:04.791011 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:04.790977 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2" Oct 29 23:55:05.094239 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:05.094214 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2"] Oct 29 23:55:05.096424 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:55:05.096396 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7bb5b36_d921_496a_8c03_953a2645aec5.slice/crio-2687d5912a33736b23eb588dfa250aafb95d48adcb77e24621f1676cb5d52696 WatchSource:0}: Error finding container 2687d5912a33736b23eb588dfa250aafb95d48adcb77e24621f1676cb5d52696: Status 404 returned error can't find the container with id 2687d5912a33736b23eb588dfa250aafb95d48adcb77e24621f1676cb5d52696 Oct 29 23:55:05.926679 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:05.926637 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2" event={"ID":"b7bb5b36-d921-496a-8c03-953a2645aec5","Type":"ContainerStarted","Data":"3b0c8cc5ece43404890a10e25455be087a100fc7bf467e16828d0a07fd17ecab"} Oct 29 23:55:05.927108 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:05.926717 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2" event={"ID":"b7bb5b36-d921-496a-8c03-953a2645aec5","Type":"ContainerStarted","Data":"2687d5912a33736b23eb588dfa250aafb95d48adcb77e24621f1676cb5d52696"} Oct 29 23:55:05.969147 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:05.969084 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2" podStartSLOduration=1.9690544399999999 podStartE2EDuration="1.96905444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:55:05.967565489 +0000 UTC m=+4352.619064022" watchObservedRunningTime="2026-10-29 23:55:05.96905444 +0000 UTC m=+4352.620552970" Oct 29 23:55:06.934481 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:06.934443 5057 generic.go:356] "Generic (PLEG): container finished" podID="b7bb5b36-d921-496a-8c03-953a2645aec5" containerID="3b0c8cc5ece43404890a10e25455be087a100fc7bf467e16828d0a07fd17ecab" exitCode=0 Oct 29 23:55:06.934937 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:06.934505 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2" event={"ID":"b7bb5b36-d921-496a-8c03-953a2645aec5","Type":"ContainerDied","Data":"3b0c8cc5ece43404890a10e25455be087a100fc7bf467e16828d0a07fd17ecab"} Oct 29 23:55:08.084821 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:08.084797 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2" Oct 29 23:55:08.126076 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:08.126036 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7bb5b36-d921-496a-8c03-953a2645aec5-kube-api-access-626pg\" (UniqueName: \"kubernetes.io/projected/b7bb5b36-d921-496a-8c03-953a2645aec5-kube-api-access-626pg\") pod \"b7bb5b36-d921-496a-8c03-953a2645aec5\" (UID: \"b7bb5b36-d921-496a-8c03-953a2645aec5\") " Oct 29 23:55:08.126282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:08.126099 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7bb5b36-d921-496a-8c03-953a2645aec5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7bb5b36-d921-496a-8c03-953a2645aec5-maas-api-xks-ca\") pod \"b7bb5b36-d921-496a-8c03-953a2645aec5\" (UID: \"b7bb5b36-d921-496a-8c03-953a2645aec5\") " Oct 29 23:55:08.126282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:08.126240 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7bb5b36-d921-496a-8c03-953a2645aec5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7bb5b36-d921-496a-8c03-953a2645aec5-maas-api-openshift-ca\") pod \"b7bb5b36-d921-496a-8c03-953a2645aec5\" (UID: \"b7bb5b36-d921-496a-8c03-953a2645aec5\") " Oct 29 23:55:08.126981 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:08.126913 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7bb5b36-d921-496a-8c03-953a2645aec5-maas-api-openshift-ca" pod "b7bb5b36-d921-496a-8c03-953a2645aec5" (UID: "b7bb5b36-d921-496a-8c03-953a2645aec5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:55:08.134407 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:08.129279 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7bb5b36-d921-496a-8c03-953a2645aec5-maas-api-xks-ca" pod "b7bb5b36-d921-496a-8c03-953a2645aec5" (UID: "b7bb5b36-d921-496a-8c03-953a2645aec5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:55:08.134407 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:08.134619 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7bb5b36-d921-496a-8c03-953a2645aec5-kube-api-access-626pg" pod "b7bb5b36-d921-496a-8c03-953a2645aec5" (UID: "b7bb5b36-d921-496a-8c03-953a2645aec5"). InnerVolumeSpecName "kube-api-access-626pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:55:08.227494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:08.227412 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7bb5b36-d921-496a-8c03-953a2645aec5-maas-api-openshift-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:08.227494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:08.227445 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-626pg\" (UniqueName: \"kubernetes.io/projected/b7bb5b36-d921-496a-8c03-953a2645aec5-kube-api-access-626pg\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:08.227494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:08.227455 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7bb5b36-d921-496a-8c03-953a2645aec5-maas-api-xks-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:08.641448 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:08.641358 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh" podUID="037f1359-4def-4ce0-a40f-4db85cb069cc" containerName="payload-processing" containerID="cri-o://7c7d9e7912b07ff09d3e05adebea53d0877a186b4f7a850211fc8f4864893ae5" gracePeriod=25 Oct 29 23:55:08.946259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:08.946222 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2" event={"ID":"b7bb5b36-d921-496a-8c03-953a2645aec5","Type":"ContainerDied","Data":"2687d5912a33736b23eb588dfa250aafb95d48adcb77e24621f1676cb5d52696"} Oct 29 23:55:08.946422 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:08.946285 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2687d5912a33736b23eb588dfa250aafb95d48adcb77e24621f1676cb5d52696" Oct 29 23:55:08.946422 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:08.946230 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271728b7-nwcx2" Oct 29 23:55:12.967954 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:12.967887 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" containerID="cri-o://97046d49c1fab966113e523587494209f22d35e85bb883299cf0ec779d08fa5d" gracePeriod=45 Oct 29 23:55:13.028237 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.028170 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": dial tcp 10.132.0.88:8000: connect: connection refused" Oct 29 23:55:13.235259 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.235230 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:55:13.271881 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.271852 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a40c346b-32d3-45d0-959f-76a53fc1083d-kube-api-access-fssgp\" (UniqueName: \"kubernetes.io/projected/a40c346b-32d3-45d0-959f-76a53fc1083d-kube-api-access-fssgp\") pod \"a40c346b-32d3-45d0-959f-76a53fc1083d\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " Oct 29 23:55:13.272069 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.271921 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-dshm\") pod \"a40c346b-32d3-45d0-959f-76a53fc1083d\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " Oct 29 23:55:13.272069 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.271950 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-tmp-dir\") pod \"a40c346b-32d3-45d0-959f-76a53fc1083d\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " Oct 29 23:55:13.272069 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.271993 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a40c346b-32d3-45d0-959f-76a53fc1083d-tls-certs\" (UniqueName: \"kubernetes.io/secret/a40c346b-32d3-45d0-959f-76a53fc1083d-tls-certs\") pod \"a40c346b-32d3-45d0-959f-76a53fc1083d\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " Oct 29 23:55:13.272281 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.272076 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-home\" (UniqueName: \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-home\") pod \"a40c346b-32d3-45d0-959f-76a53fc1083d\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " Oct 29 23:55:13.272281 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.272133 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-model-cache\") pod \"a40c346b-32d3-45d0-959f-76a53fc1083d\" (UID: \"a40c346b-32d3-45d0-959f-76a53fc1083d\") " Oct 29 23:55:13.272638 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.272615 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-model-cache" pod "a40c346b-32d3-45d0-959f-76a53fc1083d" (UID: "a40c346b-32d3-45d0-959f-76a53fc1083d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:55:13.273207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.273171 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-tmp-dir" pod "a40c346b-32d3-45d0-959f-76a53fc1083d" (UID: "a40c346b-32d3-45d0-959f-76a53fc1083d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:55:13.273891 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.273857 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-home" pod "a40c346b-32d3-45d0-959f-76a53fc1083d" (UID: "a40c346b-32d3-45d0-959f-76a53fc1083d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:55:13.275213 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.275168 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a40c346b-32d3-45d0-959f-76a53fc1083d-kube-api-access-fssgp" pod "a40c346b-32d3-45d0-959f-76a53fc1083d" (UID: "a40c346b-32d3-45d0-959f-76a53fc1083d"). InnerVolumeSpecName "kube-api-access-fssgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:55:13.279278 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.279254 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a40c346b-32d3-45d0-959f-76a53fc1083d-tls-certs" pod "a40c346b-32d3-45d0-959f-76a53fc1083d" (UID: "a40c346b-32d3-45d0-959f-76a53fc1083d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:55:13.281625 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.281553 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-dshm" pod "a40c346b-32d3-45d0-959f-76a53fc1083d" (UID: "a40c346b-32d3-45d0-959f-76a53fc1083d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:55:13.374274 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.374236 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fssgp\" (UniqueName: \"kubernetes.io/projected/a40c346b-32d3-45d0-959f-76a53fc1083d-kube-api-access-fssgp\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:13.374274 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.374267 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-dshm\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:13.374274 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.374277 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-tmp-dir\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:13.374274 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.374285 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a40c346b-32d3-45d0-959f-76a53fc1083d-tls-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:13.374666 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.374293 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-home\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:13.374666 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.374301 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a40c346b-32d3-45d0-959f-76a53fc1083d-model-cache\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:13.974802 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.974768 5057 generic.go:356] "Generic (PLEG): container finished" podID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerID="97046d49c1fab966113e523587494209f22d35e85bb883299cf0ec779d08fa5d" exitCode=0 Oct 29 23:55:13.975291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.974811 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" Oct 29 23:55:13.975291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.974859 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" event={"ID":"a40c346b-32d3-45d0-959f-76a53fc1083d","Type":"ContainerDied","Data":"97046d49c1fab966113e523587494209f22d35e85bb883299cf0ec779d08fa5d"} Oct 29 23:55:13.975291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.974907 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l" event={"ID":"a40c346b-32d3-45d0-959f-76a53fc1083d","Type":"ContainerDied","Data":"11137751ef519de163c8db39cbc9252b4471b167c06ad3195707ddeb76cc2aa5"} Oct 29 23:55:13.975291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.974935 5057 scope.go:122] "RemoveContainer" containerID="97046d49c1fab966113e523587494209f22d35e85bb883299cf0ec779d08fa5d" Oct 29 23:55:13.983766 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.983746 5057 scope.go:122] "RemoveContainer" containerID="97046d49c1fab966113e523587494209f22d35e85bb883299cf0ec779d08fa5d" Oct 29 23:55:13.984033 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:55:13.984010 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97046d49c1fab966113e523587494209f22d35e85bb883299cf0ec779d08fa5d\": container with ID starting with 97046d49c1fab966113e523587494209f22d35e85bb883299cf0ec779d08fa5d not found: ID does not exist" containerID="97046d49c1fab966113e523587494209f22d35e85bb883299cf0ec779d08fa5d" Oct 29 23:55:13.984077 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:13.984046 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97046d49c1fab966113e523587494209f22d35e85bb883299cf0ec779d08fa5d"} err="failed to get container status \"97046d49c1fab966113e523587494209f22d35e85bb883299cf0ec779d08fa5d\": rpc error: code = NotFound desc = could not find container \"97046d49c1fab966113e523587494209f22d35e85bb883299cf0ec779d08fa5d\": container with ID starting with 97046d49c1fab966113e523587494209f22d35e85bb883299cf0ec779d08fa5d not found: ID does not exist" Oct 29 23:55:14.026783 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:14.026752 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l"] Oct 29 23:55:14.038529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:14.038503 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/distinct-2-w1-99dfb0-kserve-75f949566f-znq7l"] Oct 29 23:55:14.413510 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:14.413436 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh" podUID="037f1359-4def-4ce0-a40f-4db85cb069cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:55:15.479792 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:15.479756 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" path="/var/lib/kubelet/pods/a40c346b-32d3-45d0-959f-76a53fc1083d/volumes" Oct 29 23:55:16.588426 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:16.588339 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:55:19.587585 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:19.587552 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p"] Oct 29 23:55:19.588143 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:19.587834 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" podUID="00fa2987-1fdc-4e03-a657-02118cb3b3e5" containerName="maas-api" containerID="cri-o://77a39bdfd2ff610f1898fe1ae013015c4bcf340f97c4c366bf3edf233f9f50e7" gracePeriod=30 Oct 29 23:55:19.845967 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:19.845904 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" Oct 29 23:55:19.936256 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:19.936221 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00fa2987-1fdc-4e03-a657-02118cb3b3e5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00fa2987-1fdc-4e03-a657-02118cb3b3e5-metrics-certs\") pod \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\" (UID: \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\") " Oct 29 23:55:19.936439 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:19.936278 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00fa2987-1fdc-4e03-a657-02118cb3b3e5-kube-api-access-dmdcg\" (UniqueName: \"kubernetes.io/projected/00fa2987-1fdc-4e03-a657-02118cb3b3e5-kube-api-access-dmdcg\") pod \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\" (UID: \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\") " Oct 29 23:55:19.936439 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:19.936362 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00fa2987-1fdc-4e03-a657-02118cb3b3e5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00fa2987-1fdc-4e03-a657-02118cb3b3e5-openshift-service-ca\") pod \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\" (UID: \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\") " Oct 29 23:55:19.936581 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:19.936440 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00fa2987-1fdc-4e03-a657-02118cb3b3e5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00fa2987-1fdc-4e03-a657-02118cb3b3e5-maas-api-tls\") pod \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\" (UID: \"00fa2987-1fdc-4e03-a657-02118cb3b3e5\") " Oct 29 23:55:19.936790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:19.936764 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00fa2987-1fdc-4e03-a657-02118cb3b3e5-openshift-service-ca" pod "00fa2987-1fdc-4e03-a657-02118cb3b3e5" (UID: "00fa2987-1fdc-4e03-a657-02118cb3b3e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:55:19.938546 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:19.938511 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fa2987-1fdc-4e03-a657-02118cb3b3e5-kube-api-access-dmdcg" pod "00fa2987-1fdc-4e03-a657-02118cb3b3e5" (UID: "00fa2987-1fdc-4e03-a657-02118cb3b3e5"). InnerVolumeSpecName "kube-api-access-dmdcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:55:19.938644 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:19.938553 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00fa2987-1fdc-4e03-a657-02118cb3b3e5-maas-api-tls" pod "00fa2987-1fdc-4e03-a657-02118cb3b3e5" (UID: "00fa2987-1fdc-4e03-a657-02118cb3b3e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:55:19.938810 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:19.938793 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00fa2987-1fdc-4e03-a657-02118cb3b3e5-metrics-certs" pod "00fa2987-1fdc-4e03-a657-02118cb3b3e5" (UID: "00fa2987-1fdc-4e03-a657-02118cb3b3e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:55:20.011938 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:20.011906 5057 generic.go:356] "Generic (PLEG): container finished" podID="00fa2987-1fdc-4e03-a657-02118cb3b3e5" containerID="77a39bdfd2ff610f1898fe1ae013015c4bcf340f97c4c366bf3edf233f9f50e7" exitCode=0 Oct 29 23:55:20.012110 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:20.011954 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" Oct 29 23:55:20.012110 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:20.011954 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" event={"ID":"00fa2987-1fdc-4e03-a657-02118cb3b3e5","Type":"ContainerDied","Data":"77a39bdfd2ff610f1898fe1ae013015c4bcf340f97c4c366bf3edf233f9f50e7"} Oct 29 23:55:20.012110 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:20.012081 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p" event={"ID":"00fa2987-1fdc-4e03-a657-02118cb3b3e5","Type":"ContainerDied","Data":"307d42eaed5806d198ef877c28a4e85117ede6275e6ebb95f791b29d4ff74e41"} Oct 29 23:55:20.012282 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:20.012110 5057 scope.go:122] "RemoveContainer" containerID="77a39bdfd2ff610f1898fe1ae013015c4bcf340f97c4c366bf3edf233f9f50e7" Oct 29 23:55:20.022428 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:20.022412 5057 scope.go:122] "RemoveContainer" containerID="77a39bdfd2ff610f1898fe1ae013015c4bcf340f97c4c366bf3edf233f9f50e7" Oct 29 23:55:20.022743 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:55:20.022722 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a39bdfd2ff610f1898fe1ae013015c4bcf340f97c4c366bf3edf233f9f50e7\": container with ID starting with 77a39bdfd2ff610f1898fe1ae013015c4bcf340f97c4c366bf3edf233f9f50e7 not found: ID does not exist" containerID="77a39bdfd2ff610f1898fe1ae013015c4bcf340f97c4c366bf3edf233f9f50e7" Oct 29 23:55:20.022818 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:20.022752 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a39bdfd2ff610f1898fe1ae013015c4bcf340f97c4c366bf3edf233f9f50e7"} err="failed to get container status \"77a39bdfd2ff610f1898fe1ae013015c4bcf340f97c4c366bf3edf233f9f50e7\": rpc error: code = NotFound desc = could not find container \"77a39bdfd2ff610f1898fe1ae013015c4bcf340f97c4c366bf3edf233f9f50e7\": container with ID starting with 77a39bdfd2ff610f1898fe1ae013015c4bcf340f97c4c366bf3edf233f9f50e7 not found: ID does not exist" Oct 29 23:55:20.037469 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:20.037451 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00fa2987-1fdc-4e03-a657-02118cb3b3e5-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:20.037555 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:20.037471 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00fa2987-1fdc-4e03-a657-02118cb3b3e5-metrics-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:20.037555 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:20.037480 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmdcg\" (UniqueName: \"kubernetes.io/projected/00fa2987-1fdc-4e03-a657-02118cb3b3e5-kube-api-access-dmdcg\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:20.037555 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:20.037490 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00fa2987-1fdc-4e03-a657-02118cb3b3e5-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:20.064666 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:20.064629 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p"] Oct 29 23:55:20.069038 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:20.069009 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-99dfb0-84f64bc848-z6t8p"] Oct 29 23:55:21.479908 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:21.479872 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00fa2987-1fdc-4e03-a657-02118cb3b3e5" path="/var/lib/kubelet/pods/00fa2987-1fdc-4e03-a657-02118cb3b3e5/volumes" Oct 29 23:55:23.186800 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:23.186768 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xhhgl"] Oct 29 23:55:24.413779 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:24.413745 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh" podUID="037f1359-4def-4ce0-a40f-4db85cb069cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:55:28.654794 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:28.654737 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-fbgn7_08ce15dc-71d7-4b48-9ad4-75a826414293/manager/0.log" Oct 29 23:55:28.841852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:28.841822 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-j2gvf_cabd04f5-8346-495d-9f50-6c38a692df10/kuadrant-console-plugin/0.log" Oct 29 23:55:28.896040 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:28.896010 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh" Oct 29 23:55:29.033473 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:29.033445 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/037f1359-4def-4ce0-a40f-4db85cb069cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037f1359-4def-4ce0-a40f-4db85cb069cc-plugins-config-volume\") pod \"037f1359-4def-4ce0-a40f-4db85cb069cc\" (UID: \"037f1359-4def-4ce0-a40f-4db85cb069cc\") " Oct 29 23:55:29.033694 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:29.033576 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/037f1359-4def-4ce0-a40f-4db85cb069cc-kube-api-access-rjfhv\" (UniqueName: \"kubernetes.io/projected/037f1359-4def-4ce0-a40f-4db85cb069cc-kube-api-access-rjfhv\") pod \"037f1359-4def-4ce0-a40f-4db85cb069cc\" (UID: \"037f1359-4def-4ce0-a40f-4db85cb069cc\") " Oct 29 23:55:29.033876 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:29.033851 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/037f1359-4def-4ce0-a40f-4db85cb069cc-plugins-config-volume" pod "037f1359-4def-4ce0-a40f-4db85cb069cc" (UID: "037f1359-4def-4ce0-a40f-4db85cb069cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:55:29.034093 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:29.034041 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037f1359-4def-4ce0-a40f-4db85cb069cc-plugins-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:29.035828 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:29.035792 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037f1359-4def-4ce0-a40f-4db85cb069cc-kube-api-access-rjfhv" pod "037f1359-4def-4ce0-a40f-4db85cb069cc" (UID: "037f1359-4def-4ce0-a40f-4db85cb069cc"). InnerVolumeSpecName "kube-api-access-rjfhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:55:29.062103 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:29.062064 5057 generic.go:356] "Generic (PLEG): container finished" podID="037f1359-4def-4ce0-a40f-4db85cb069cc" containerID="7c7d9e7912b07ff09d3e05adebea53d0877a186b4f7a850211fc8f4864893ae5" exitCode=0 Oct 29 23:55:29.062301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:29.062108 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh" Oct 29 23:55:29.062301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:29.062150 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh" event={"ID":"037f1359-4def-4ce0-a40f-4db85cb069cc","Type":"ContainerDied","Data":"7c7d9e7912b07ff09d3e05adebea53d0877a186b4f7a850211fc8f4864893ae5"} Oct 29 23:55:29.062301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:29.062243 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh" event={"ID":"037f1359-4def-4ce0-a40f-4db85cb069cc","Type":"ContainerDied","Data":"0304daca905eec59c755d533b4496cff713af56e6b22e18db2b3e30129963e55"} Oct 29 23:55:29.062301 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:29.062275 5057 scope.go:122] "RemoveContainer" containerID="7c7d9e7912b07ff09d3e05adebea53d0877a186b4f7a850211fc8f4864893ae5" Oct 29 23:55:29.072930 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:29.072907 5057 scope.go:122] "RemoveContainer" containerID="7c7d9e7912b07ff09d3e05adebea53d0877a186b4f7a850211fc8f4864893ae5" Oct 29 23:55:29.073302 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:55:29.073275 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c7d9e7912b07ff09d3e05adebea53d0877a186b4f7a850211fc8f4864893ae5\": container with ID starting with 7c7d9e7912b07ff09d3e05adebea53d0877a186b4f7a850211fc8f4864893ae5 not found: ID does not exist" containerID="7c7d9e7912b07ff09d3e05adebea53d0877a186b4f7a850211fc8f4864893ae5" Oct 29 23:55:29.073402 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:29.073314 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c7d9e7912b07ff09d3e05adebea53d0877a186b4f7a850211fc8f4864893ae5"} err="failed to get container status \"7c7d9e7912b07ff09d3e05adebea53d0877a186b4f7a850211fc8f4864893ae5\": rpc error: code = NotFound desc = could not find container \"7c7d9e7912b07ff09d3e05adebea53d0877a186b4f7a850211fc8f4864893ae5\": container with ID starting with 7c7d9e7912b07ff09d3e05adebea53d0877a186b4f7a850211fc8f4864893ae5 not found: ID does not exist" Oct 29 23:55:29.118778 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:29.118741 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh"] Oct 29 23:55:29.120544 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:29.120520 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-99dfb0-6f86598d4-xhhgh"] Oct 29 23:55:29.135263 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:29.135229 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjfhv\" (UniqueName: \"kubernetes.io/projected/037f1359-4def-4ce0-a40f-4db85cb069cc-kube-api-access-rjfhv\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:29.210176 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:29.210126 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zm2vt_55c4bb94-26cc-45e9-ac4c-3542b1758fcb/manager/1.log" Oct 29 23:55:29.430622 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:29.430579 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xhhgl_7b0ec9a4-4f75-4119-9744-364ee7d69731/limitador/0.log" Oct 29 23:55:29.479773 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:29.479741 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="037f1359-4def-4ce0-a40f-4db85cb069cc" path="/var/lib/kubelet/pods/037f1359-4def-4ce0-a40f-4db85cb069cc/volumes" Oct 29 23:55:29.626643 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:29.626611 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-gmpzb_a58ed6fc-270f-4377-83b4-486019dbe952/manager/0.log" Oct 29 23:55:30.454849 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:30.454809 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zm2vt_55c4bb94-26cc-45e9-ac4c-3542b1758fcb/manager/0.log" Oct 29 23:55:33.912535 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:55:33.912493 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:55:42.263915 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:42.263879 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df9d4f548-fhxv7_c748534c-ac5c-4bd0-98a0-51dc11412f9c/manager/1.log" Oct 29 23:55:43.519052 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:43.519018 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df9d4f548-fhxv7_c748534c-ac5c-4bd0-98a0-51dc11412f9c/manager/0.log" Oct 29 23:55:53.694887 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:53.694848 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw"] Oct 29 23:55:53.695498 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:53.695065 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" podUID="88b5b01f-807f-4fa6-b4aa-b1c338e2e3df" containerName="discovery" containerID="cri-o://7771cc1187adf7c5ce7b83c538425f60a21ad8a614f9c614a7c39d4bcaf9f93b" gracePeriod=30 Oct 29 23:55:53.956813 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:53.956745 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 23:55:54.055617 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.055582 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-kube-api-access-znwxp\" (UniqueName: \"kubernetes.io/projected/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-kube-api-access-znwxp\") pod \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " Oct 29 23:55:54.055825 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.055630 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-csr-ca-configmap\") pod \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " Oct 29 23:55:54.055825 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.055667 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-csr-dns-cert\") pod \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " Oct 29 23:55:54.055825 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.055702 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-kubeconfig\") pod \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " Oct 29 23:55:54.055825 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.055752 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-token\" (UniqueName: \"kubernetes.io/projected/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-token\") pod \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " Oct 29 23:55:54.055825 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.055776 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-local-certs\") pod \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " Oct 29 23:55:54.055825 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.055805 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-cacerts\" (UniqueName: \"kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-cacerts\") pod \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\" (UID: \"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df\") " Oct 29 23:55:54.056228 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.056062 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-csr-ca-configmap" pod "88b5b01f-807f-4fa6-b4aa-b1c338e2e3df" (UID: "88b5b01f-807f-4fa6-b4aa-b1c338e2e3df"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:55:54.058033 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.058008 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-kubeconfig" pod "88b5b01f-807f-4fa6-b4aa-b1c338e2e3df" (UID: "88b5b01f-807f-4fa6-b4aa-b1c338e2e3df"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:55:54.058218 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.058169 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-kube-api-access-znwxp" pod "88b5b01f-807f-4fa6-b4aa-b1c338e2e3df" (UID: "88b5b01f-807f-4fa6-b4aa-b1c338e2e3df"). InnerVolumeSpecName "kube-api-access-znwxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:55:54.058339 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.058319 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-csr-dns-cert" pod "88b5b01f-807f-4fa6-b4aa-b1c338e2e3df" (UID: "88b5b01f-807f-4fa6-b4aa-b1c338e2e3df"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:55:54.058401 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.058380 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-cacerts" pod "88b5b01f-807f-4fa6-b4aa-b1c338e2e3df" (UID: "88b5b01f-807f-4fa6-b4aa-b1c338e2e3df"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:55:54.058463 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.058442 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-local-certs" pod "88b5b01f-807f-4fa6-b4aa-b1c338e2e3df" (UID: "88b5b01f-807f-4fa6-b4aa-b1c338e2e3df"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:55:54.058515 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.058495 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-token" pod "88b5b01f-807f-4fa6-b4aa-b1c338e2e3df" (UID: "88b5b01f-807f-4fa6-b4aa-b1c338e2e3df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:55:54.157285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.157249 5057 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-kubeconfig\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:54.157285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.157278 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:54.157285 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.157287 5057 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-local-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:54.157530 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.157298 5057 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-cacerts\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:54.157530 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.157307 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znwxp\" (UniqueName: \"kubernetes.io/projected/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-kube-api-access-znwxp\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:54.157530 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.157316 5057 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-csr-ca-configmap\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:54.157530 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.157325 5057 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df-istio-csr-dns-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Oct 29 23:55:54.197597 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.197559 5057 generic.go:356] "Generic (PLEG): container finished" podID="88b5b01f-807f-4fa6-b4aa-b1c338e2e3df" containerID="7771cc1187adf7c5ce7b83c538425f60a21ad8a614f9c614a7c39d4bcaf9f93b" exitCode=0 Oct 29 23:55:54.197790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.197610 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" Oct 29 23:55:54.197790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.197650 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" event={"ID":"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df","Type":"ContainerDied","Data":"7771cc1187adf7c5ce7b83c538425f60a21ad8a614f9c614a7c39d4bcaf9f93b"} Oct 29 23:55:54.197790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.197739 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw" event={"ID":"88b5b01f-807f-4fa6-b4aa-b1c338e2e3df","Type":"ContainerDied","Data":"68262d7ed1ba6ec1da67fe324c443feaba5853146ad787c910c82dc889aabe75"} Oct 29 23:55:54.197790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.197766 5057 scope.go:122] "RemoveContainer" containerID="7771cc1187adf7c5ce7b83c538425f60a21ad8a614f9c614a7c39d4bcaf9f93b" Oct 29 23:55:54.216582 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.216563 5057 scope.go:122] "RemoveContainer" containerID="7771cc1187adf7c5ce7b83c538425f60a21ad8a614f9c614a7c39d4bcaf9f93b" Oct 29 23:55:54.216856 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:55:54.216838 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7771cc1187adf7c5ce7b83c538425f60a21ad8a614f9c614a7c39d4bcaf9f93b\": container with ID starting with 7771cc1187adf7c5ce7b83c538425f60a21ad8a614f9c614a7c39d4bcaf9f93b not found: ID does not exist" containerID="7771cc1187adf7c5ce7b83c538425f60a21ad8a614f9c614a7c39d4bcaf9f93b" Oct 29 23:55:54.216908 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.216863 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7771cc1187adf7c5ce7b83c538425f60a21ad8a614f9c614a7c39d4bcaf9f93b"} err="failed to get container status \"7771cc1187adf7c5ce7b83c538425f60a21ad8a614f9c614a7c39d4bcaf9f93b\": rpc error: code = NotFound desc = could not find container \"7771cc1187adf7c5ce7b83c538425f60a21ad8a614f9c614a7c39d4bcaf9f93b\": container with ID starting with 7771cc1187adf7c5ce7b83c538425f60a21ad8a614f9c614a7c39d4bcaf9f93b not found: ID does not exist" Oct 29 23:55:54.249527 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.249502 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw"] Oct 29 23:55:54.253605 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:54.253585 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vf9zw"] Oct 29 23:55:55.478993 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:55:55.478962 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88b5b01f-807f-4fa6-b4aa-b1c338e2e3df" path="/var/lib/kubelet/pods/88b5b01f-807f-4fa6-b4aa-b1c338e2e3df/volumes" Oct 29 23:56:03.911617 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:56:03.911571 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:56:05.732857 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:05.732821 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df9d4f548-fhxv7_c748534c-ac5c-4bd0-98a0-51dc11412f9c/manager/1.log" Oct 29 23:56:06.975726 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:06.975694 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df9d4f548-fhxv7_c748534c-ac5c-4bd0-98a0-51dc11412f9c/manager/0.log" Oct 29 23:56:07.763167 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:07.763138 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-271728b7-nwcx2_b7bb5b36-d921-496a-8c03-953a2645aec5/revoke-keys/0.log" Oct 29 23:56:08.817136 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:08.817102 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-df20bf02-fzvcb_c82f8e25-d512-4f0d-9148-5ae241c0424b/revoke-keys/0.log" Oct 29 23:56:11.277078 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:11.277044 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-fbgn7_08ce15dc-71d7-4b48-9ad4-75a826414293/manager/0.log" Oct 29 23:56:11.455962 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:11.455934 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-j2gvf_cabd04f5-8346-495d-9f50-6c38a692df10/kuadrant-console-plugin/0.log" Oct 29 23:56:11.789140 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:11.789093 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zm2vt_55c4bb94-26cc-45e9-ac4c-3542b1758fcb/manager/1.log" Oct 29 23:56:12.031174 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:12.031141 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xhhgl_7b0ec9a4-4f75-4119-9744-364ee7d69731/limitador/0.log" Oct 29 23:56:12.217072 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:12.217040 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-gmpzb_a58ed6fc-270f-4377-83b4-486019dbe952/manager/0.log" Oct 29 23:56:13.023590 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:13.023516 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zm2vt_55c4bb94-26cc-45e9-ac4c-3542b1758fcb/manager/0.log" Oct 29 23:56:24.750673 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:24.750638 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf_12f9734d-d6d4-4a4e-a0b1-13ebd9505d54/istio-proxy/0.log" Oct 29 23:56:25.328437 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:25.328409 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7945df5b5-pcgp6_99577b75-29cd-430b-b5ac-2af3828586d0/router/0.log" Oct 29 23:56:26.497089 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:26.497015 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-2qkxv_907c36dd-72ba-42b8-86f1-9ff84400b4b2/main/0.log" Oct 29 23:56:26.888884 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:26.888810 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-7ks2l_bce69276-30ac-47e6-99df-81608424b3dd/main/0.log" Oct 29 23:56:27.276008 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:27.275978 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-5m4cw_5f7503b3-a2b7-4795-b78a-9f18d00b5f2f/main/0.log" Oct 29 23:56:33.911530 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:56:33.911488 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:56:57.821025 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.820988 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z7kg9/must-gather-fnbnt"] Oct 29 23:56:57.821852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.821596 5057 cpu_manager.go:414] "containerMap: removing container" podUID="88b5b01f-807f-4fa6-b4aa-b1c338e2e3df" containerName="discovery" Oct 29 23:56:57.821852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.821617 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88b5b01f-807f-4fa6-b4aa-b1c338e2e3df" containerName="discovery" Oct 29 23:56:57.821852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.821636 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b7bb5b36-d921-496a-8c03-953a2645aec5" containerName="revoke-keys" Oct 29 23:56:57.821852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.821643 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7bb5b36-d921-496a-8c03-953a2645aec5" containerName="revoke-keys" Oct 29 23:56:57.821852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.821668 5057 cpu_manager.go:414] "containerMap: removing container" podUID="037f1359-4def-4ce0-a40f-4db85cb069cc" containerName="payload-processing" Oct 29 23:56:57.821852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.821677 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="037f1359-4def-4ce0-a40f-4db85cb069cc" containerName="payload-processing" Oct 29 23:56:57.821852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.821687 5057 cpu_manager.go:414] "containerMap: removing container" podUID="408b4a17-ac05-47f5-bbcb-ed398229570c" containerName="istio-proxy" Oct 29 23:56:57.821852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.821694 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408b4a17-ac05-47f5-bbcb-ed398229570c" containerName="istio-proxy" Oct 29 23:56:57.821852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.821707 5057 cpu_manager.go:414] "containerMap: removing container" podUID="00fa2987-1fdc-4e03-a657-02118cb3b3e5" containerName="maas-api" Oct 29 23:56:57.821852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.821714 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00fa2987-1fdc-4e03-a657-02118cb3b3e5" containerName="maas-api" Oct 29 23:56:57.821852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.821733 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" Oct 29 23:56:57.821852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.821740 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" Oct 29 23:56:57.821852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.821759 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f7feb618-94bf-4988-abca-4dc4fbd699c7" containerName="istio-proxy" Oct 29 23:56:57.821852 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.821766 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7feb618-94bf-4988-abca-4dc4fbd699c7" containerName="istio-proxy" Oct 29 23:56:57.822628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.821877 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="88b5b01f-807f-4fa6-b4aa-b1c338e2e3df" containerName="discovery" Oct 29 23:56:57.822628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.821888 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7bb5b36-d921-496a-8c03-953a2645aec5" containerName="revoke-keys" Oct 29 23:56:57.822628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.821900 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="037f1359-4def-4ce0-a40f-4db85cb069cc" containerName="payload-processing" Oct 29 23:56:57.822628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.821911 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="00fa2987-1fdc-4e03-a657-02118cb3b3e5" containerName="maas-api" Oct 29 23:56:57.822628 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.821924 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a40c346b-32d3-45d0-959f-76a53fc1083d" containerName="main" Oct 29 23:56:57.825582 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.825561 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z7kg9/must-gather-fnbnt" Oct 29 23:56:57.846424 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.846400 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z7kg9/must-gather-fnbnt"] Oct 29 23:56:57.856463 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.856438 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z7kg9\"/\"openshift-service-ca.crt\"" Oct 29 23:56:57.856686 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.856667 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z7kg9\"/\"kube-root-ca.crt\"" Oct 29 23:56:57.856752 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.856737 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z7kg9\"/\"default-dockercfg-8svfs\"" Oct 29 23:56:57.890968 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.890940 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t99pv/must-gather-nxs6q"] Oct 29 23:56:57.894650 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.894630 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wjxqn/must-gather-78g9r"] Oct 29 23:56:57.894779 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.894762 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t99pv/must-gather-nxs6q" Oct 29 23:56:57.898042 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.898021 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wjxqn/must-gather-78g9r" Oct 29 23:56:57.919917 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.919894 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t99pv/must-gather-nxs6q"] Oct 29 23:56:57.922153 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.922131 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wjxqn/must-gather-78g9r"] Oct 29 23:56:57.943973 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.943948 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/db2bfae0-a114-45d4-a52d-b40a3a9538af-must-gather-output\") pod \"must-gather-fnbnt\" (UID: \"db2bfae0-a114-45d4-a52d-b40a3a9538af\") " pod="openshift-must-gather-z7kg9/must-gather-fnbnt" Oct 29 23:56:57.944121 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.944021 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tpf5\" (UniqueName: \"kubernetes.io/projected/db2bfae0-a114-45d4-a52d-b40a3a9538af-kube-api-access-5tpf5\") pod \"must-gather-fnbnt\" (UID: \"db2bfae0-a114-45d4-a52d-b40a3a9538af\") " pod="openshift-must-gather-z7kg9/must-gather-fnbnt" Oct 29 23:56:57.945291 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.945263 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wjxqn\"/\"kube-root-ca.crt\"" Oct 29 23:56:57.945422 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.945332 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t99pv\"/\"kube-root-ca.crt\"" Oct 29 23:56:57.945422 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.945390 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t99pv\"/\"openshift-service-ca.crt\"" Oct 29 23:56:57.955704 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.955687 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t99pv\"/\"default-dockercfg-8lgdb\"" Oct 29 23:56:57.955956 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.955941 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wjxqn\"/\"openshift-service-ca.crt\"" Oct 29 23:56:57.956009 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:57.955971 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wjxqn\"/\"default-dockercfg-kjcnv\"" Oct 29 23:56:58.044819 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.044787 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhpqr\" (UniqueName: \"kubernetes.io/projected/1a8b3dea-a756-467a-bc94-45c452c306d2-kube-api-access-mhpqr\") pod \"must-gather-78g9r\" (UID: \"1a8b3dea-a756-467a-bc94-45c452c306d2\") " pod="openshift-must-gather-wjxqn/must-gather-78g9r" Oct 29 23:56:58.044968 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.044836 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1a8b3dea-a756-467a-bc94-45c452c306d2-must-gather-output\") pod \"must-gather-78g9r\" (UID: \"1a8b3dea-a756-467a-bc94-45c452c306d2\") " pod="openshift-must-gather-wjxqn/must-gather-78g9r" Oct 29 23:56:58.044968 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.044924 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/db2bfae0-a114-45d4-a52d-b40a3a9538af-must-gather-output\") pod \"must-gather-fnbnt\" (UID: \"db2bfae0-a114-45d4-a52d-b40a3a9538af\") " pod="openshift-must-gather-z7kg9/must-gather-fnbnt" Oct 29 23:56:58.045053 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.044968 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tpf5\" (UniqueName: \"kubernetes.io/projected/db2bfae0-a114-45d4-a52d-b40a3a9538af-kube-api-access-5tpf5\") pod \"must-gather-fnbnt\" (UID: \"db2bfae0-a114-45d4-a52d-b40a3a9538af\") " pod="openshift-must-gather-z7kg9/must-gather-fnbnt" Oct 29 23:56:58.045053 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.045012 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2rtr\" (UniqueName: \"kubernetes.io/projected/80357469-7071-47bc-a2ae-a705efc1c15f-kube-api-access-r2rtr\") pod \"must-gather-nxs6q\" (UID: \"80357469-7071-47bc-a2ae-a705efc1c15f\") " pod="openshift-must-gather-t99pv/must-gather-nxs6q" Oct 29 23:56:58.045053 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.045041 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/80357469-7071-47bc-a2ae-a705efc1c15f-must-gather-output\") pod \"must-gather-nxs6q\" (UID: \"80357469-7071-47bc-a2ae-a705efc1c15f\") " pod="openshift-must-gather-t99pv/must-gather-nxs6q" Oct 29 23:56:58.045319 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.045303 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/db2bfae0-a114-45d4-a52d-b40a3a9538af-must-gather-output\") pod \"must-gather-fnbnt\" (UID: \"db2bfae0-a114-45d4-a52d-b40a3a9538af\") " pod="openshift-must-gather-z7kg9/must-gather-fnbnt" Oct 29 23:56:58.070511 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.070483 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tpf5\" (UniqueName: \"kubernetes.io/projected/db2bfae0-a114-45d4-a52d-b40a3a9538af-kube-api-access-5tpf5\") pod \"must-gather-fnbnt\" (UID: \"db2bfae0-a114-45d4-a52d-b40a3a9538af\") " pod="openshift-must-gather-z7kg9/must-gather-fnbnt" Oct 29 23:56:58.139840 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.139771 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z7kg9/must-gather-fnbnt" Oct 29 23:56:58.146080 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.146057 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1a8b3dea-a756-467a-bc94-45c452c306d2-must-gather-output\") pod \"must-gather-78g9r\" (UID: \"1a8b3dea-a756-467a-bc94-45c452c306d2\") " pod="openshift-must-gather-wjxqn/must-gather-78g9r" Oct 29 23:56:58.146202 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.146121 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2rtr\" (UniqueName: \"kubernetes.io/projected/80357469-7071-47bc-a2ae-a705efc1c15f-kube-api-access-r2rtr\") pod \"must-gather-nxs6q\" (UID: \"80357469-7071-47bc-a2ae-a705efc1c15f\") " pod="openshift-must-gather-t99pv/must-gather-nxs6q" Oct 29 23:56:58.146202 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.146147 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/80357469-7071-47bc-a2ae-a705efc1c15f-must-gather-output\") pod \"must-gather-nxs6q\" (UID: \"80357469-7071-47bc-a2ae-a705efc1c15f\") " pod="openshift-must-gather-t99pv/must-gather-nxs6q" Oct 29 23:56:58.146336 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.146202 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhpqr\" (UniqueName: \"kubernetes.io/projected/1a8b3dea-a756-467a-bc94-45c452c306d2-kube-api-access-mhpqr\") pod \"must-gather-78g9r\" (UID: \"1a8b3dea-a756-467a-bc94-45c452c306d2\") " pod="openshift-must-gather-wjxqn/must-gather-78g9r" Oct 29 23:56:58.146460 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.146441 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1a8b3dea-a756-467a-bc94-45c452c306d2-must-gather-output\") pod \"must-gather-78g9r\" (UID: \"1a8b3dea-a756-467a-bc94-45c452c306d2\") " pod="openshift-must-gather-wjxqn/must-gather-78g9r" Oct 29 23:56:58.146540 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.146520 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/80357469-7071-47bc-a2ae-a705efc1c15f-must-gather-output\") pod \"must-gather-nxs6q\" (UID: \"80357469-7071-47bc-a2ae-a705efc1c15f\") " pod="openshift-must-gather-t99pv/must-gather-nxs6q" Oct 29 23:56:58.169377 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.169353 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2rtr\" (UniqueName: \"kubernetes.io/projected/80357469-7071-47bc-a2ae-a705efc1c15f-kube-api-access-r2rtr\") pod \"must-gather-nxs6q\" (UID: \"80357469-7071-47bc-a2ae-a705efc1c15f\") " pod="openshift-must-gather-t99pv/must-gather-nxs6q" Oct 29 23:56:58.169482 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.169410 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhpqr\" (UniqueName: \"kubernetes.io/projected/1a8b3dea-a756-467a-bc94-45c452c306d2-kube-api-access-mhpqr\") pod \"must-gather-78g9r\" (UID: \"1a8b3dea-a756-467a-bc94-45c452c306d2\") " pod="openshift-must-gather-wjxqn/must-gather-78g9r" Oct 29 23:56:58.209509 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.209480 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t99pv/must-gather-nxs6q" Oct 29 23:56:58.214714 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.214556 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wjxqn/must-gather-78g9r" Oct 29 23:56:58.460210 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.460169 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z7kg9/must-gather-fnbnt"] Oct 29 23:56:58.466079 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:56:58.466055 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb2bfae0_a114_45d4_a52d_b40a3a9538af.slice/crio-30b5c44338fa45afb967dc4a9722fe7ec97add81194200262c5e0f78cfc2183e WatchSource:0}: Error finding container 30b5c44338fa45afb967dc4a9722fe7ec97add81194200262c5e0f78cfc2183e: Status 404 returned error can't find the container with id 30b5c44338fa45afb967dc4a9722fe7ec97add81194200262c5e0f78cfc2183e Oct 29 23:56:58.468403 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.468382 5057 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:56:58.511278 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.511243 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z7kg9/must-gather-fnbnt" event={"ID":"db2bfae0-a114-45d4-a52d-b40a3a9538af","Type":"ContainerStarted","Data":"30b5c44338fa45afb967dc4a9722fe7ec97add81194200262c5e0f78cfc2183e"} Oct 29 23:56:58.539790 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.539771 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t99pv/must-gather-nxs6q"] Oct 29 23:56:58.541644 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:56:58.541615 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80357469_7071_47bc_a2ae_a705efc1c15f.slice/crio-c5e53a21dc886c51b2517abade7fec4bb1d54440973190835422b78d9c770551 WatchSource:0}: Error finding container c5e53a21dc886c51b2517abade7fec4bb1d54440973190835422b78d9c770551: Status 404 returned error can't find the container with id c5e53a21dc886c51b2517abade7fec4bb1d54440973190835422b78d9c770551 Oct 29 23:56:58.547395 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:56:58.547370 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a8b3dea_a756_467a_bc94_45c452c306d2.slice/crio-3201b21395d74eaa9710e0f0ac24b34924acd0310dd6634acfab4a000d954511 WatchSource:0}: Error finding container 3201b21395d74eaa9710e0f0ac24b34924acd0310dd6634acfab4a000d954511: Status 404 returned error can't find the container with id 3201b21395d74eaa9710e0f0ac24b34924acd0310dd6634acfab4a000d954511 Oct 29 23:56:58.548880 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:58.548860 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wjxqn/must-gather-78g9r"] Oct 29 23:56:59.543851 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:59.543799 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t99pv/must-gather-nxs6q" event={"ID":"80357469-7071-47bc-a2ae-a705efc1c15f","Type":"ContainerStarted","Data":"c5e53a21dc886c51b2517abade7fec4bb1d54440973190835422b78d9c770551"} Oct 29 23:56:59.571054 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:56:59.571014 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjxqn/must-gather-78g9r" event={"ID":"1a8b3dea-a756-467a-bc94-45c452c306d2","Type":"ContainerStarted","Data":"3201b21395d74eaa9710e0f0ac24b34924acd0310dd6634acfab4a000d954511"} Oct 29 23:57:00.588494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:00.588400 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t99pv/must-gather-nxs6q" event={"ID":"80357469-7071-47bc-a2ae-a705efc1c15f","Type":"ContainerStarted","Data":"744addbafcfd434bb9e40d7c30ddf2602f0d950a51c3a39341e257428992730c"} Oct 29 23:57:00.588494 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:00.588455 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t99pv/must-gather-nxs6q" event={"ID":"80357469-7071-47bc-a2ae-a705efc1c15f","Type":"ContainerStarted","Data":"2c197bcd149959b18fb4353cb9350773ad1a3fbb32f2a75fd7b9367f9a426a8e"} Oct 29 23:57:00.632493 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:00.632272 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-t99pv/must-gather-nxs6q" podStartSLOduration=2.468694922 podStartE2EDuration="3.632248531s" totalImagesPullingTime="1.163553609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:57:00.630069998 +0000 UTC m=+4467.281568532" watchObservedRunningTime="2026-10-29 23:57:00.632248531 +0000 UTC m=+4467.283747064" Oct 29 23:57:03.840329 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:03.840294 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b2cf2_41ea529e-ab5d-419c-87bf-fe0bd1a6460f/global-pull-secret-syncer/0.log" Oct 29 23:57:03.921354 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:57:03.921311 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:57:04.383541 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:04.383180 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-7hz4v_6a4e4c73-0516-464c-9b6e-1c4adf93c03b/connection-checker/0.log" Oct 29 23:57:04.499313 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:04.499276 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-fh7rx_a1a232f4-f67c-404f-a1f7-af0c3cc30180/connection-checker/0.log" Oct 29 23:57:04.616281 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:04.616252 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-k8xnq_68daf26a-81a0-4bd5-a188-9b39a1f460a7/connection-checker/0.log" Oct 29 23:57:05.081990 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:05.078452 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sc55n_2d0ad66c-ca75-48ef-aa6b-89e0d2fb2db9/konnectivity-agent/0.log" Oct 29 23:57:05.466207 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:05.456743 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-80.ec2.internal_4239c6df062516f41cb9c5984a7633c8/haproxy/0.log" Oct 29 23:57:12.311711 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:12.310858 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-fbgn7_08ce15dc-71d7-4b48-9ad4-75a826414293/manager/0.log" Oct 29 23:57:12.431077 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:12.431044 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-j2gvf_cabd04f5-8346-495d-9f50-6c38a692df10/kuadrant-console-plugin/0.log" Oct 29 23:57:12.977096 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:12.977061 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zm2vt_55c4bb94-26cc-45e9-ac4c-3542b1758fcb/manager/1.log" Oct 29 23:57:13.274039 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:13.273580 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zm2vt_55c4bb94-26cc-45e9-ac4c-3542b1758fcb/manager/0.log" Oct 29 23:57:13.377462 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:13.377376 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xhhgl_7b0ec9a4-4f75-4119-9744-364ee7d69731/limitador/0.log" Oct 29 23:57:13.576541 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:13.576264 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-gmpzb_a58ed6fc-270f-4377-83b4-486019dbe952/manager/0.log" Oct 29 23:57:14.822215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:14.820721 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjxqn/must-gather-78g9r" event={"ID":"1a8b3dea-a756-467a-bc94-45c452c306d2","Type":"ContainerStarted","Data":"00bab1f70bfc585c228850f4599b8a59a8ec3ecb3e38ac5304fcac0f86fd0698"} Oct 29 23:57:14.822215 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:14.820784 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjxqn/must-gather-78g9r" event={"ID":"1a8b3dea-a756-467a-bc94-45c452c306d2","Type":"ContainerStarted","Data":"c195b7039bb3fb8f3df1248247049ad366544d90ac0a1d6174e3e431e96c2d23"} Oct 29 23:57:14.826246 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:14.826211 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z7kg9/must-gather-fnbnt" event={"ID":"db2bfae0-a114-45d4-a52d-b40a3a9538af","Type":"ContainerStarted","Data":"224fb8f7f1ea984cb36dcc7fd34299ffce6766ecc9ccaef0aa8c3c1dc7d88bbb"} Oct 29 23:57:14.826486 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:14.826463 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z7kg9/must-gather-fnbnt" event={"ID":"db2bfae0-a114-45d4-a52d-b40a3a9538af","Type":"ContainerStarted","Data":"53cf02024290602825c6fd6d372283f206c47243ff5eec80508da5da14ceb1f1"} Oct 29 23:57:14.910265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:14.909378 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-z7kg9/must-gather-fnbnt" podStartSLOduration=2.661786377 podStartE2EDuration="17.909362567s" totalImagesPullingTime="15.24757619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:57:14.907482264 +0000 UTC m=+4481.558980795" watchObservedRunningTime="2026-10-29 23:57:14.909362567 +0000 UTC m=+4481.560861096" Oct 29 23:57:14.910265 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:14.909979 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wjxqn/must-gather-78g9r" podStartSLOduration=2.400029372 podStartE2EDuration="17.909970952s" totalImagesPullingTime="15.50994158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:57:14.865897219 +0000 UTC m=+4481.517395752" watchObservedRunningTime="2026-10-29 23:57:14.909970952 +0000 UTC m=+4481.561469489" Oct 29 23:57:16.009388 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.009036 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq"] Oct 29 23:57:16.017212 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.015298 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" Oct 29 23:57:16.040950 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.040914 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq"] Oct 29 23:57:16.162693 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.162650 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1860a4b5-2be2-40e4-8c59-562b2ad78d7a-lib-modules\") pod \"perf-node-gather-daemonset-mftnq\" (UID: \"1860a4b5-2be2-40e4-8c59-562b2ad78d7a\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" Oct 29 23:57:16.162870 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.162749 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1860a4b5-2be2-40e4-8c59-562b2ad78d7a-podres\") pod \"perf-node-gather-daemonset-mftnq\" (UID: \"1860a4b5-2be2-40e4-8c59-562b2ad78d7a\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" Oct 29 23:57:16.162870 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.162796 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kh9r\" (UniqueName: \"kubernetes.io/projected/1860a4b5-2be2-40e4-8c59-562b2ad78d7a-kube-api-access-2kh9r\") pod \"perf-node-gather-daemonset-mftnq\" (UID: \"1860a4b5-2be2-40e4-8c59-562b2ad78d7a\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" Oct 29 23:57:16.162870 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.162842 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1860a4b5-2be2-40e4-8c59-562b2ad78d7a-proc\") pod \"perf-node-gather-daemonset-mftnq\" (UID: \"1860a4b5-2be2-40e4-8c59-562b2ad78d7a\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" Oct 29 23:57:16.163045 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.162906 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1860a4b5-2be2-40e4-8c59-562b2ad78d7a-sys\") pod \"perf-node-gather-daemonset-mftnq\" (UID: \"1860a4b5-2be2-40e4-8c59-562b2ad78d7a\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" Oct 29 23:57:16.217208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.216858 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-dzx6b_ad00084a-5a9f-489b-9122-2316f617f3f1/cluster-monitoring-operator/0.log" Oct 29 23:57:16.264130 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.263995 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1860a4b5-2be2-40e4-8c59-562b2ad78d7a-podres\") pod \"perf-node-gather-daemonset-mftnq\" (UID: \"1860a4b5-2be2-40e4-8c59-562b2ad78d7a\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" Oct 29 23:57:16.264130 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.264063 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kh9r\" (UniqueName: \"kubernetes.io/projected/1860a4b5-2be2-40e4-8c59-562b2ad78d7a-kube-api-access-2kh9r\") pod \"perf-node-gather-daemonset-mftnq\" (UID: \"1860a4b5-2be2-40e4-8c59-562b2ad78d7a\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" Oct 29 23:57:16.264130 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.264102 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1860a4b5-2be2-40e4-8c59-562b2ad78d7a-proc\") pod \"perf-node-gather-daemonset-mftnq\" (UID: \"1860a4b5-2be2-40e4-8c59-562b2ad78d7a\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" Oct 29 23:57:16.264484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.264158 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1860a4b5-2be2-40e4-8c59-562b2ad78d7a-sys\") pod \"perf-node-gather-daemonset-mftnq\" (UID: \"1860a4b5-2be2-40e4-8c59-562b2ad78d7a\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" Oct 29 23:57:16.264484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.264243 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1860a4b5-2be2-40e4-8c59-562b2ad78d7a-lib-modules\") pod \"perf-node-gather-daemonset-mftnq\" (UID: \"1860a4b5-2be2-40e4-8c59-562b2ad78d7a\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" Oct 29 23:57:16.264484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.264405 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1860a4b5-2be2-40e4-8c59-562b2ad78d7a-lib-modules\") pod \"perf-node-gather-daemonset-mftnq\" (UID: \"1860a4b5-2be2-40e4-8c59-562b2ad78d7a\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" Oct 29 23:57:16.264484 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.264474 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1860a4b5-2be2-40e4-8c59-562b2ad78d7a-podres\") pod \"perf-node-gather-daemonset-mftnq\" (UID: \"1860a4b5-2be2-40e4-8c59-562b2ad78d7a\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" Oct 29 23:57:16.264834 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.264810 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1860a4b5-2be2-40e4-8c59-562b2ad78d7a-sys\") pod \"perf-node-gather-daemonset-mftnq\" (UID: \"1860a4b5-2be2-40e4-8c59-562b2ad78d7a\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" Oct 29 23:57:16.264905 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.264813 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1860a4b5-2be2-40e4-8c59-562b2ad78d7a-proc\") pod \"perf-node-gather-daemonset-mftnq\" (UID: \"1860a4b5-2be2-40e4-8c59-562b2ad78d7a\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" Oct 29 23:57:16.291636 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.291597 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kh9r\" (UniqueName: \"kubernetes.io/projected/1860a4b5-2be2-40e4-8c59-562b2ad78d7a-kube-api-access-2kh9r\") pod \"perf-node-gather-daemonset-mftnq\" (UID: \"1860a4b5-2be2-40e4-8c59-562b2ad78d7a\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" Oct 29 23:57:16.362382 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.362352 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" Oct 29 23:57:16.696012 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.695982 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq"] Oct 29 23:57:16.698443 ip-10-0-135-80 kubenswrapper[5057]: W1029 23:57:16.698400 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1860a4b5_2be2_40e4_8c59_562b2ad78d7a.slice/crio-3f12ea1dfa6369e18e9b1ed2f5fee9e42220f1e27186dfa783df53844e02b372 WatchSource:0}: Error finding container 3f12ea1dfa6369e18e9b1ed2f5fee9e42220f1e27186dfa783df53844e02b372: Status 404 returned error can't find the container with id 3f12ea1dfa6369e18e9b1ed2f5fee9e42220f1e27186dfa783df53844e02b372 Oct 29 23:57:16.856781 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:16.847427 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" event={"ID":"1860a4b5-2be2-40e4-8c59-562b2ad78d7a","Type":"ContainerStarted","Data":"3f12ea1dfa6369e18e9b1ed2f5fee9e42220f1e27186dfa783df53844e02b372"} Oct 29 23:57:17.374377 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:17.374349 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mrtf7_225b1337-fa33-4932-97c0-bbad4f83aab4/node-exporter/0.log" Oct 29 23:57:17.492965 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:17.492937 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mrtf7_225b1337-fa33-4932-97c0-bbad4f83aab4/kube-rbac-proxy/0.log" Oct 29 23:57:17.617024 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:17.616984 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mrtf7_225b1337-fa33-4932-97c0-bbad4f83aab4/init-textfile/0.log" Oct 29 23:57:17.854337 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:17.854295 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" event={"ID":"1860a4b5-2be2-40e4-8c59-562b2ad78d7a","Type":"ContainerStarted","Data":"0834fc5481effa4e04724d83bc5fe4864e186f2bdbdcb1b9c379540a09e727a5"} Oct 29 23:57:17.854543 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:17.854483 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" Oct 29 23:57:17.896700 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:17.896633 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" podStartSLOduration=2.896617386 podStartE2EDuration="2.896617386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:57:17.895393744 +0000 UTC m=+4484.546892279" watchObservedRunningTime="2026-10-29 23:57:17.896617386 +0000 UTC m=+4484.548115918" Oct 29 23:57:18.389537 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:18.389508 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df9d4f548-fhxv7_c748534c-ac5c-4bd0-98a0-51dc11412f9c/manager/0.log" Oct 29 23:57:18.584934 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:18.584881 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df9d4f548-fhxv7_c748534c-ac5c-4bd0-98a0-51dc11412f9c/manager/0.log" Oct 29 23:57:18.965419 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:18.965377 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df9d4f548-fhxv7_c748534c-ac5c-4bd0-98a0-51dc11412f9c/manager/1.log" Oct 29 23:57:19.179890 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:19.179841 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df9d4f548-fhxv7_c748534c-ac5c-4bd0-98a0-51dc11412f9c/manager/1.log" Oct 29 23:57:19.578390 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:19.578358 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-582ld_16d0a575-91ff-47c7-ab38-a62c542e4c9c/prometheus-operator-admission-webhook/0.log" Oct 29 23:57:22.285735 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:22.285700 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-rwfmh_575ee3c4-7c77-4b31-9961-eebc60254d19/networking-console-plugin/0.log" Oct 29 23:57:23.434601 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:23.434569 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:57:23.440611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:23.440573 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/2.log" Oct 29 23:57:23.593223 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:23.592778 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df9d4f548-fhxv7_c748534c-ac5c-4bd0-98a0-51dc11412f9c/manager/0.log" Oct 29 23:57:23.647515 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:23.647483 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df9d4f548-fhxv7_c748534c-ac5c-4bd0-98a0-51dc11412f9c/manager/0.log" Oct 29 23:57:23.880529 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:23.880450 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mftnq" Oct 29 23:57:24.320571 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:24.319552 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df9d4f548-fhxv7_c748534c-ac5c-4bd0-98a0-51dc11412f9c/manager/1.log" Oct 29 23:57:24.362611 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:24.362576 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df9d4f548-fhxv7_c748534c-ac5c-4bd0-98a0-51dc11412f9c/manager/1.log" Oct 29 23:57:24.747092 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:24.747057 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-cbgxh_cb6f2831-3e8c-440c-a5da-327ab5cf947b/download-server/0.log" Oct 29 23:57:26.276031 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:26.275967 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-868dt_8f6c01b9-e044-413c-b533-09033d04f29e/volume-data-source-validator/0.log" Oct 29 23:57:29.191372 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:29.191338 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gj6vs_266bff54-6027-4822-91ce-be5cbd316ddc/dns/0.log" Oct 29 23:57:29.281793 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:29.281732 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df9d4f548-fhxv7_c748534c-ac5c-4bd0-98a0-51dc11412f9c/manager/0.log" Oct 29 23:57:29.307313 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:29.307104 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gj6vs_266bff54-6027-4822-91ce-be5cbd316ddc/kube-rbac-proxy/0.log" Oct 29 23:57:29.377880 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:29.377838 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df9d4f548-fhxv7_c748534c-ac5c-4bd0-98a0-51dc11412f9c/manager/0.log" Oct 29 23:57:29.677814 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:29.677780 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5czm4_af926a18-1147-4786-8bd9-8a6727abb6fa/dns-node-resolver/0.log" Oct 29 23:57:29.989974 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:29.989889 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df9d4f548-fhxv7_c748534c-ac5c-4bd0-98a0-51dc11412f9c/manager/1.log" Oct 29 23:57:30.063615 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:30.063577 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df9d4f548-fhxv7_c748534c-ac5c-4bd0-98a0-51dc11412f9c/manager/1.log" Oct 29 23:57:31.348843 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:31.348801 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-585b5456bf-pk6zv_33110848-7553-4685-b74a-08ef49fa172e/registry/0.log" Oct 29 23:57:31.687097 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:31.687058 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rwz4x_4525bd80-124f-4894-9fc0-b8235a0fe689/node-ca/0.log" Oct 29 23:57:33.917880 ip-10-0-135-80 kubenswrapper[5057]: E1029 23:57:33.917837 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6rqj6" containerName="extractor" Oct 29 23:57:34.193355 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:34.193321 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf_12f9734d-d6d4-4a4e-a0b1-13ebd9505d54/istio-proxy/0.log" Oct 29 23:57:34.388675 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:34.388637 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7945df5b5-pcgp6_99577b75-29cd-430b-b5ac-2af3828586d0/router/0.log" Oct 29 23:57:35.494208 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:35.490307 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:57:35.511778 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:35.511606 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lm55w_9e91a303-9b80-47f5-a502-fbb6365eaa54/console-operator/1.log" Oct 29 23:57:36.166116 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:36.166073 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf_12f9734d-d6d4-4a4e-a0b1-13ebd9505d54/istio-proxy/0.log" Oct 29 23:57:36.281870 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:36.281820 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-645b5cfbff-6bcgf_12f9734d-d6d4-4a4e-a0b1-13ebd9505d54/istio-proxy/0.log" Oct 29 23:57:36.368683 ip-10-0-135-80 kubenswrapper[5057]: I1029 23:57:36.368628 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7945df5b5-pcgp6_99577b75-29cd-430b-b5ac-2af3828586d0/router/0.log"